* [RFC] using lightweight containers instead of chroot
@ 2021-07-08 9:07 Cedric Hombourger
2021-07-08 11:38 ` Jan Kiszka
2021-07-26 13:55 ` Anton Mikanovich
0 siblings, 2 replies; 13+ messages in thread
From: Cedric Hombourger @ 2021-07-08 9:07 UTC (permalink / raw)
To: isar-users; +Cc: MacDonald, Joe
Hello,
This is a RFC for us to replace uses of chroot within the Isar classes
to use Linux namespaces.
Motivation
We will be submitting another RFC for per-recipe buildchroots but we
should note that it would likely
increase the number of active bind mounts during our builds. So
before we discuss that, we may want to
discuss how we run our build scripts today using chroots.
There are numerous challenges with our current implementation and
attempts to , Examples would include:
1e31d5a events: Warn if mounted paths left
d21d495 dpkg: Make mount buildroot reliable
22c42de dpkg-base: Warn about unmounting problems
...
And you have found yourself in the situation where Isar wiped out
/dev or your layers (at least I did)
chroot is a powerful tool but it is starting to show its age and we
see how much burden is on us to setup and
shutdown our chroot environment
Proposal
We may want to use unshare(1) to create a mount namespace where we
will create our bind mounts,
chroot into the buildchroot and run the specified command/script
The immediate benefit of this approach is that all mounts
automatically disappear as the supplied
command exits (whether it aborts prematurely because of an error or
normally on completion).
Another nice benefit is that bind mounts we created within this
namespace are not (directly) visible
from the parent namespace
However, we found that running scripts within an unshare environment
may not be as easy as
chroot. We would welcome feedback on the code snippets provided
below if you happen to have
some better ideas.
def isar_user_spec():
import os
return '%d:%d' % (os.getuid(), os.getgid())
ISAR_USER_SPEC = "${@ isar_user_spec() }"
ISAR_UNSHARE_CMD = "sudo unshare --pid --fork --ipc --mount sh -ex"
ISAR_CHROOT_SHELL = "sh -ex"
ISAR_CHROOT_ROOT = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
ISAR_CHROOT_USER = "chroot --userspec='${ISAR_USER_SPEC}'
${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
# Would be similar to buildchroot_do_mounts but will happen in a
separate mount namespace
BUILDCHROOT_DO_MOUNTS =
" \
mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
'${BUILDCHROOT_DIR}/isar-apt' ; \
mount --bind '${DL_DIR}'
'${BUILDCHROOT_DIR}/downloads' ; \
mount -t proc none
'${BUILDCHROOT_DIR}/proc' ; \
mount --rbind /sys
'${BUILDCHROOT_DIR}/sys' ; \
mount -t tmpfs -o rw,nosuid,nodev,seclabel none
${BUILDCHROOT_DIR}/dev/shm ; \
mount -o bind /dev/pts
${BUILDCHROOT_DIR}/dev/pts \
"
# Would be similar to dpkg_do_mounts but will happen in a separate
mount namespace
DPKG_DO_MOUNTS = " \
${BUILDCHROOT_DO_MOUNTS} ; \
mkdir -p ${BUILDROOT} ; \
mount --bind ${WORKDIR} ${BUILDROOT} \
"
# Build package from sources using build script
_runbuild() {
export arch=${1}
E="${@ isar_export_proxies(d)}"
( cat <<" UNSHARE"
${DPKG_DO_MOUNTS}
( cat <<" SCRIPT"
export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
/isar/build.sh ${PP}/${PPS} ${arch}
SCRIPT
) | ${ISAR_CHROOT_USER}
UNSHARE
) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
}
dpkg_runbuild() {
( _runbuild ${PACKAGE_ARCH} )
}
PS: I am not very happy with the need to feed the script to execute
under unshare
via stdin, if there are better ways, we would be happy to
consider them!
Proposed Next steps
1. Collect feedback and answer questions
a. is the use of unshare a good idea? (no is an OK answer!)
b. can we come up with a better code construct?
2. Check compatibility with containerized-builds (e.g. from within a
kas build)
3. Check for better ways to spawn scripts
4. Implement and submit RFC patches
Future RFCs
- per-recipe buildchroots (mimic sbuild)
Cedric
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 9:07 [RFC] using lightweight containers instead of chroot Cedric Hombourger
@ 2021-07-08 11:38 ` Jan Kiszka
2021-07-08 13:52 ` Helmut Grohne
2021-07-26 13:55 ` Anton Mikanovich
1 sibling, 1 reply; 13+ messages in thread
From: Jan Kiszka @ 2021-07-08 11:38 UTC (permalink / raw)
To: Cedric Hombourger, isar-users, Baurzhan Ismagulov, Helmut Grohne
Cc: MacDonald, Joe
On 08.07.21 11:07, Cedric Hombourger wrote:
> Hello,
>
> This is a RFC for us to replace uses of chroot within the Isar classes
> to use Linux namespaces.
>
> Motivation
>
> We will be submitting another RFC for per-recipe buildchroots but we
Good that you are sharing this: Baurzhan and his team are currently
working on sbuilder support with separate buildchroots. The idea of
using sbuilder is to get closer to how Debian builds packages as well.
Longterm, there is also the desire to include support for DPKG_ROOT as
chroot-less way of building packages, faster when doing it cross and
also without special permissions (e.g. for qemu-user). But that requires
per-package support from Debian upstream. Discussions only started, in
particular with Helmuth (added to CC).
> should note that it would likely
> increase the number of active bind mounts during our builds. So
> before we discuss that, we may want to
> discuss how we run our build scripts today using chroots.
>
> There are numerous challenges with our current implementation and
> attempts to , Examples would include:
>
> 1e31d5a events: Warn if mounted paths left
> d21d495 dpkg: Make mount buildroot reliable
> 22c42de dpkg-base: Warn about unmounting problems
> ...
>
> And you have found yourself in the situation where Isar wiped out
> /dev or your layers (at least I did)
>
> chroot is a powerful tool but it is starting to show its age and we
> see how much burden is on us to setup and
> shutdown our chroot environment
>
> Proposal
>
> We may want to use unshare(1) to create a mount namespace where we
> will create our bind mounts,
> chroot into the buildchroot and run the specified command/script
>
> The immediate benefit of this approach is that all mounts
> automatically disappear as the supplied
> command exits (whether it aborts prematurely because of an error or
> normally on completion).
>
> Another nice benefit is that bind mounts we created within this
> namespace are not (directly) visible
> from the parent namespace
>
> However, we found that running scripts within an unshare environment
> may not be as easy as
> chroot. We would welcome feedback on the code snippets provided
> below if you happen to have
> some better ideas.
I suspect Helmuth can tell us if that would take us on a fragile path
from Debian perspective. Isar-internal implementation details we could
likely sort out, but if that approach has architectural limits /wrt what
Debian packages expect/require, it might be the wrong direction.
>
> def isar_user_spec():
> import os
> return '%d:%d' % (os.getuid(), os.getgid())
>
> ISAR_USER_SPEC = "${@ isar_user_spec() }"
> ISAR_UNSHARE_CMD = "sudo unshare --pid --fork --ipc --mount sh -ex"
> ISAR_CHROOT_SHELL = "sh -ex"
> ISAR_CHROOT_ROOT = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
> ISAR_CHROOT_USER = "chroot --userspec='${ISAR_USER_SPEC}'
> ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>
> # Would be similar to buildchroot_do_mounts but will happen in a
> separate mount namespace
> BUILDCHROOT_DO_MOUNTS =
> " \
> mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
> '${BUILDCHROOT_DIR}/isar-apt' ; \
> mount --bind '${DL_DIR}'
> '${BUILDCHROOT_DIR}/downloads' ; \
> mount -t proc none
> '${BUILDCHROOT_DIR}/proc' ; \
> mount --rbind /sys
> '${BUILDCHROOT_DIR}/sys' ; \
> mount -t tmpfs -o rw,nosuid,nodev,seclabel none
> ${BUILDCHROOT_DIR}/dev/shm ; \
> mount -o bind /dev/pts
> ${BUILDCHROOT_DIR}/dev/pts \
> "
>
> # Would be similar to dpkg_do_mounts but will happen in a separate
> mount namespace
> DPKG_DO_MOUNTS = " \
> ${BUILDCHROOT_DO_MOUNTS} ; \
> mkdir -p ${BUILDROOT} ; \
> mount --bind ${WORKDIR} ${BUILDROOT} \
> "
>
> # Build package from sources using build script
> _runbuild() {
> export arch=${1}
>
> E="${@ isar_export_proxies(d)}"
> ( cat <<" UNSHARE"
> ${DPKG_DO_MOUNTS}
> ( cat <<" SCRIPT"
> export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
> export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
> export PARALLEL_MAKE="${PARALLEL_MAKE}"
> /isar/build.sh ${PP}/${PPS} ${arch}
> SCRIPT
> ) | ${ISAR_CHROOT_USER}
> UNSHARE
> ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
> }
>
> dpkg_runbuild() {
> ( _runbuild ${PACKAGE_ARCH} )
> }
>
> PS: I am not very happy with the need to feed the script to execute
> under unshare
> via stdin, if there are better ways, we would be happy to
> consider them!
>
> Proposed Next steps
>
> 1. Collect feedback and answer questions
> a. is the use of unshare a good idea? (no is an OK answer!)
> b. can we come up with a better code construct?
>
> 2. Check compatibility with containerized-builds (e.g. from within a
> kas build)
>
That would obviously be a critical thing, compatibility with both
classic Docker and podman.
> 3. Check for better ways to spawn scripts
>
> 4. Implement and submit RFC patches
>
> Future RFCs
>
> - per-recipe buildchroots (mimic sbuild)
>
Please coordinate here who could work on what and who already has
completed something so that we do not end up with two implementations
but rather one that is even better.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 11:38 ` Jan Kiszka
@ 2021-07-08 13:52 ` Helmut Grohne
2021-07-08 17:16 ` Jan Kiszka
2021-07-09 15:46 ` Cedric Hombourger
0 siblings, 2 replies; 13+ messages in thread
From: Helmut Grohne @ 2021-07-08 13:52 UTC (permalink / raw)
To: Jan Kiszka
Cc: Cedric Hombourger, isar-users, Baurzhan Ismagulov, MacDonald, Joe
On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
> On 08.07.21 11:07, Cedric Hombourger wrote:
> ...
> Longterm, there is also the desire to include support for DPKG_ROOT as
> chroot-less way of building packages, faster when doing it cross and
> also without special permissions (e.g. for qemu-user). But that requires
> per-package support from Debian upstream. Discussions only started, in
> particular with Helmuth (added to CC).
You appear to be confusing some aspects here. DPKG_ROOT is not relevant
to building packages. It is only relevant to installing them (which may
be relevant here for creating filesystem images).
Building packages without chroot in a reproducible way seems next to
impossible to me. Even when you use user namespaces, chroot does not go
away. It merely becomes unprivileged. Is that what you mean here?
> > Proposal
> >
> > �� We may want to use unshare(1) to create a mount namespace where we
> > �� will create our bind mounts,
> > �� chroot into the buildchroot and run the specified command/script
Are you aware that sbuild directly supports this use case? It has a
--mode argument and one of its values is "unshare". In that case, you
supply a tarball containing the chroot and it'll perform an unprivileged
build inside an unshared chroot.
> > �� The immediate benefit of this approach is that all mounts
> > �� automatically disappear as the supplied
> > �� command exits (whether it aborts prematurely because of an error or
> > �� normally on completion).
> >
> > �� Another nice benefit is that bind mounts we created within this
> > �� namespace are not (directly) visible
> > �� from the parent namespace
> >
> > �� However, we found that running scripts within an unshare environment
> > �� may not be as easy as
> > �� chroot. We would welcome feedback on the code snippets provided
> > �� below if you happen to have
> > �� some better ideas.
All of this applies to sbuild --mode=unshare as well except that it
makes running scripts from hooks simple.
> I suspect Helmuth can tell us if that would take us on a fragile path
> from Debian perspective. Isar-internal implementation details we could
> likely sort out, but if that approach has architectural limits /wrt what
> Debian packages expect/require, it might be the wrong direction.
Reimplementing this functionality seems like a waste of time to me. If
we ignore that for a moment, we notice that there are already ~10
implementations and updating them all is painful. Therefore, we can
conclude that changes to the build environment are rare and your
reimplementation likely is maintainable with limited effort.
> >
> > �� def isar_user_spec():
> > ��� ��� import os
> > ��� ��� return '%d:%d' % (os.getuid(), os.getgid())
> >
> > �� ISAR_USER_SPEC��� = "${@ isar_user_spec() }"
> > �� ISAR_UNSHARE_CMD� = "sudo unshare --pid --fork --ipc --mount sh -ex"
> > �� ISAR_CHROOT_SHELL = "sh -ex"
> > �� ISAR_CHROOT_ROOT� = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
> > �� ISAR_CHROOT_USER� = "chroot --userspec='${ISAR_USER_SPEC}'
> > �� ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
> >
> > �� # Would be similar to buildchroot_do_mounts but will happen in a
> > �� separate mount namespace
> > �� BUILDCHROOT_DO_MOUNTS =
> > �� "�������������������������������������������������������� \
> > ��� ��� mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
> > �� '${BUILDCHROOT_DIR}/isar-apt'���� ; \
> > ��� ��� mount --bind '${DL_DIR}'
> > �� '${BUILDCHROOT_DIR}/downloads'�������������������� ; \
> > ��� ��� mount -t proc none
> > �� '${BUILDCHROOT_DIR}/proc'������������������������������� ; \
> > ��� ��� mount --rbind /sys
> > �� '${BUILDCHROOT_DIR}/sys'�������������������������������� ; \
> > ��� ��� mount -t tmpfs -o rw,nosuid,nodev,seclabel none
> > �� ${BUILDCHROOT_DIR}/dev/shm� ; \
> > ��� ��� mount -o bind /dev/pts
> > �� ${BUILDCHROOT_DIR}/dev/pts���������������������������� \
> > �� "
> >
> > �� # Would be similar to dpkg_do_mounts but will happen in a separate
> > �� mount namespace
> > �� DPKG_DO_MOUNTS = "������������������������ \
> > ��� ��� ${BUILDCHROOT_DO_MOUNTS}������������ ; \
> > ��� ��� mkdir -p ${BUILDROOT}��������������� ; \
> > ��� ��� mount --bind ${WORKDIR} ${BUILDROOT}�� \
> > �� "
> >
> > �� # Build package from sources using build script
> > �� _runbuild() {
> > ��� ��� export arch=${1}
> >
> > ��� ��� E="${@ isar_export_proxies(d)}"
> > ��� ��� (�� cat <<"������� UNSHARE"
> > ��� ����������� ${DPKG_DO_MOUNTS}
> > ��� ����������� (�� cat <<"��������������� SCRIPT"
> > ��� ������������������� export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
> > ��� ������������������� export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
> > ��� ������������������� export PARALLEL_MAKE="${PARALLEL_MAKE}"
> > ��� ������������������� /isar/build.sh ${PP}/${PPS} ${arch}
> > ��� ��������������� SCRIPT
> > ��� ����������� ) | ${ISAR_CHROOT_USER}
> > ��� ������� UNSHARE
> > ��� ��� ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
> > �� }
> >
> > �� dpkg_runbuild() {
> > ��� ��� ( _runbuild ${PACKAGE_ARCH} )
> > �� }
> >
> > �� PS: I am not very happy with the need to feed the script to execute
> > �� under unshare
> > ��� ��� via stdin, if there are better ways, we would be happy to
> > �� consider them!
When I started talking to Jan, I proposed adding an abstraction layer
for package building. Work on that layer has now progressed under the
name "mdbp" and source is available at
https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
Raphael Hertzog on unifying the API with debusine. Let us for a moment
consider the implications of using mdbp here.
* Much of the complexity would go away. What you are left with is
writing a json file describing how you want your package built. What
gets a little more difficult is getting the isar-apt repository past
mdbp. Likely that would require a temporary http server.
* mdbp is not another builder, but an adapter to existing ones. It can
perform your builds using an existing sbuild or pbuilder
installation. If you want more isolation, maybe using debspawn
(backed by systemd-nspawn) is for you?
* mdbp also provides a stateless backend that uses mmdebstrap. This
backend performs the build in a user namespace.
* If you decide that you prefer building in docker, we can add a
backend for e.g. debocker or something else.
I'm not sure what you'd be missing by using mdbp here, but one thing
you'd certainly miss is quite a bit of complex code.
If you want to consider this route, read the schema first:
https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json
Helmut
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 13:52 ` Helmut Grohne
@ 2021-07-08 17:16 ` Jan Kiszka
2021-07-08 19:34 ` Helmut Grohne
2021-07-09 15:46 ` Cedric Hombourger
1 sibling, 1 reply; 13+ messages in thread
From: Jan Kiszka @ 2021-07-08 17:16 UTC (permalink / raw)
To: Helmut Grohne
Cc: Cedric Hombourger, isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 08.07.21 15:52, Helmut Grohne wrote:
> On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
>> On 08.07.21 11:07, Cedric Hombourger wrote:
>> ...
>
>> Longterm, there is also the desire to include support for DPKG_ROOT as
>> chroot-less way of building packages, faster when doing it cross and
>> also without special permissions (e.g. for qemu-user). But that requires
>> per-package support from Debian upstream. Discussions only started, in
>> particular with Helmuth (added to CC).
>
> You appear to be confusing some aspects here. DPKG_ROOT is not relevant
> to building packages. It is only relevant to installing them (which may
> be relevant here for creating filesystem images).
>
> Building packages without chroot in a reproducible way seems next to
> impossible to me. Even when you use user namespaces, chroot does not go
> away. It merely becomes unprivileged. Is that what you mean here?
Yeah, not cleanly separated: We need DPKG_ROOT to install build
dependencies unprivileged into a build environment, and then we need
something ideally unprivileged - e.g. namespaces - to run the build
inside that environment.
>
>>> Proposal
>>>
>>> We may want to use unshare(1) to create a mount namespace where we
>>> will create our bind mounts,
>>> chroot into the buildchroot and run the specified command/script
>
> Are you aware that sbuild directly supports this use case? It has a
> --mode argument and one of its values is "unshare". In that case, you
> supply a tarball containing the chroot and it'll perform an unprivileged
> build inside an unshared chroot.
Possibly, the prototype on Ilbers side is already using that? I didn't
see it yet, just speculating.
>
>>> The immediate benefit of this approach is that all mounts
>>> automatically disappear as the supplied
>>> command exits (whether it aborts prematurely because of an error or
>>> normally on completion).
>>>
>>> Another nice benefit is that bind mounts we created within this
>>> namespace are not (directly) visible
>>> from the parent namespace
>>>
>>> However, we found that running scripts within an unshare environment
>>> may not be as easy as
>>> chroot. We would welcome feedback on the code snippets provided
>>> below if you happen to have
>>> some better ideas.
>
> All of this applies to sbuild --mode=unshare as well except that it
> makes running scripts from hooks simple.
>
>> I suspect Helmuth can tell us if that would take us on a fragile path
>> from Debian perspective. Isar-internal implementation details we could
>> likely sort out, but if that approach has architectural limits /wrt what
>> Debian packages expect/require, it might be the wrong direction.
>
> Reimplementing this functionality seems like a waste of time to me. If
> we ignore that for a moment, we notice that there are already ~10
> implementations and updating them all is painful. Therefore, we can
> conclude that changes to the build environment are rare and your
> reimplementation likely is maintainable with limited effort.
>
We don't want to reimplement if key requirements can be fulfilled with
Debian's own tools. One of them is having control over how build
dependencies are installed and from where (use cases: isar-apt, offline
builds). Another one would be the ability to patch an original package
after unpacking its sources.
>>>
>>> def isar_user_spec():
>>> import os
>>> return '%d:%d' % (os.getuid(), os.getgid())
>>>
>>> ISAR_USER_SPEC = "${@ isar_user_spec() }"
>>> ISAR_UNSHARE_CMD = "sudo unshare --pid --fork --ipc --mount sh -ex"
>>> ISAR_CHROOT_SHELL = "sh -ex"
>>> ISAR_CHROOT_ROOT = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>> ISAR_CHROOT_USER = "chroot --userspec='${ISAR_USER_SPEC}'
>>> ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>
>>> # Would be similar to buildchroot_do_mounts but will happen in a
>>> separate mount namespace
>>> BUILDCHROOT_DO_MOUNTS =
>>> " \
>>> mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>>> '${BUILDCHROOT_DIR}/isar-apt' ; \
>>> mount --bind '${DL_DIR}'
>>> '${BUILDCHROOT_DIR}/downloads' ; \
>>> mount -t proc none
>>> '${BUILDCHROOT_DIR}/proc' ; \
>>> mount --rbind /sys
>>> '${BUILDCHROOT_DIR}/sys' ; \
>>> mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>>> ${BUILDCHROOT_DIR}/dev/shm ; \
>>> mount -o bind /dev/pts
>>> ${BUILDCHROOT_DIR}/dev/pts \
>>> "
>>>
>>> # Would be similar to dpkg_do_mounts but will happen in a separate
>>> mount namespace
>>> DPKG_DO_MOUNTS = " \
>>> ${BUILDCHROOT_DO_MOUNTS} ; \
>>> mkdir -p ${BUILDROOT} ; \
>>> mount --bind ${WORKDIR} ${BUILDROOT} \
>>> "
>>>
>>> # Build package from sources using build script
>>> _runbuild() {
>>> export arch=${1}
>>>
>>> E="${@ isar_export_proxies(d)}"
>>> ( cat <<" UNSHARE"
>>> ${DPKG_DO_MOUNTS}
>>> ( cat <<" SCRIPT"
>>> export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>>> export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>>> export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>> /isar/build.sh ${PP}/${PPS} ${arch}
>>> SCRIPT
>>> ) | ${ISAR_CHROOT_USER}
>>> UNSHARE
>>> ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>>> }
>>>
>>> dpkg_runbuild() {
>>> ( _runbuild ${PACKAGE_ARCH} )
>>> }
>>>
>>> PS: I am not very happy with the need to feed the script to execute
>>> under unshare
>>> via stdin, if there are better ways, we would be happy to
>>> consider them!
>
> When I started talking to Jan, I proposed adding an abstraction layer
> for package building. Work on that layer has now progressed under the
> name "mdbp" and source is available at
> https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
> Raphael Hertzog on unifying the API with debusine. Let us for a moment
> consider the implications of using mdbp here.
>
> * Much of the complexity would go away. What you are left with is
> writing a json file describing how you want your package built. What
> gets a little more difficult is getting the isar-apt repository past
> mdbp. Likely that would require a temporary http server.
Why can't that also support a local folder with a apt repo? Spawning
servers is surely doable but can become fun (unless using containers...)
when running multiple build in parallel.
> * mdbp is not another builder, but an adapter to existing ones. It can
> perform your builds using an existing sbuild or pbuilder
> installation. If you want more isolation, maybe using debspawn
> (backed by systemd-nspawn) is for you?
> * mdbp also provides a stateless backend that uses mmdebstrap. This
> backend performs the build in a user namespace.
If that is not going to be rebuilt for every package (cached baseline,
some dependency versions for all built packages in a run) and could
possibly also be reused for other things (running the imaging), it could
be a building block to replace our buildchroots. If not, we would have
to continue feeding it with separately bootstrapped chroots.
> * If you decide that you prefer building in docker, we can add a
> backend for e.g. debocker or something else.
Most of the Isar builds (complete images, not just packages) are already
happening inside containers (kas-isar). Thus we rather need to be
careful not requiring any nesting that could become complex - at best.
At the same time, we want Isar to continue work on a plain Debian system.
>
> I'm not sure what you'd be missing by using mdbp here, but one thing
> you'd certainly miss is quite a bit of complex code.
>
> If you want to consider this route, read the schema first:
> https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json
>
The key requirements are listed above. Baurzhan looked into mdbp
already, maybe he can comment on further details.
The key question for us remains what value the abstraction can have for
Isar, today and on the long run. Switching builders is not a daily
business so far. Isar needs one, and that needs to plug well into the
image creation workflow. If interacting with mdbp can be simpler than
talking to sbuild directly, that would be a clear gain. If it's rather
more complex under the Isar constraints while we do not have the
compelling use case for switching builders, it is likely not the best
way forward.
Thanks,
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 17:16 ` Jan Kiszka
@ 2021-07-08 19:34 ` Helmut Grohne
2021-07-12 8:25 ` Jan Kiszka
0 siblings, 1 reply; 13+ messages in thread
From: Helmut Grohne @ 2021-07-08 19:34 UTC (permalink / raw)
To: Jan Kiszka
Cc: Cedric Hombourger, isar-users, Baurzhan Ismagulov, MacDonald, Joe
Hi Jan,
On Thu, Jul 08, 2021 at 07:16:05PM +0200, Jan Kiszka wrote:
> Yeah, not cleanly separated: We need DPKG_ROOT to install build
> dependencies unprivileged into a build environment, and then we need
> something ideally unprivileged - e.g. namespaces - to run the build
> inside that environment.
Still wrong. DPKG_ROOT is not meant to install build dependencies. I
doubt that it covers sufficient packages to do so any time soon. I also
don't see any point in doing so when you can just chroot (unprivileged
via user namespaces) into it and install them there.
The reason to want DPKG_ROOT is to prepare the root filesystem for your
embedded board. If that's not what you want to do, DPKG_ROOT is not your
tool.
> We don't want to reimplement if key requirements can be fulfilled with
> Debian's own tools. One of them is having control over how build
> dependencies are installed and from where (use cases: isar-apt, offline
> builds). Another one would be the ability to patch an original package
> after unpacking its sources.
I think the goal should be making Debian's tools fulfil your key
requirements. (Regardless of whether using an abstraction layer or not.)
Can you elaborate how you want to control your build dependencies? It
would seem to me that you could just patch debian/control and be done,
no? What else do you need to control? Do you want to change the
resolver?
The "from where" part seems relatively easy to me. All of the build
tools support one way or another to supply extra repositories. All you
need to do here is ensure that your extra packages from extra
repositories have a higher version than those in your base distribution.
Why does your build tool need to support patching after unpacking
sources? I'm using that approach in rebootstrap and I consider it a
design mistake. You can just extract your source package outside the
builder, bump the version, apply your patches, create a new source
package and build that. Creating actual source packages saves you from a
lot of trouble.
> Why can't that also support a local folder with a apt repo? Spawning
> servers is surely doable but can become fun (unless using containers...)
> when running multiple build in parallel.
I've considered that, but support for this is lacking at this point.
Ideally, you'd want to bind mount the local folder into the chroot. When
using user namespaces your mapped root user may be underprivileged to
see that local folder already. sbuild does not provide such means
(unless you modify the underlying schroot, which requires root).
pbuilder has --bindmounts (but doesn't use a user namespace). mmdebstrap
can only sync directories, which means copying them.
> If that is not going to be rebuilt for every package (cached baseline,
> some dependency versions for all built packages in a run) and could
> possibly also be reused for other things (running the imaging), it could
> be a building block to replace our buildchroots. If not, we would have
> to continue feeding it with separately bootstrapped chroots.
Where does that "not rebuilt" requirement come from? Is that for
performance reasons?
If yes, please benchmark it. I've locally compared build times
(sbuild/pbuilder/mmdebstrap) and the difference is small (<10 seconds).
After all, extracting a pbuilder (or sbuild) base.tgz is not immediate
either and mmdebstrap is quite fast.
> Most of the Isar builds (complete images, not just packages) are already
> happening inside containers (kas-isar). Thus we rather need to be
> careful not requiring any nesting that could become complex - at best.
> At the same time, we want Isar to continue work on a plain Debian system.
You cannot use user namespaces in secure docker containers anyway, but
then when you already are inside a container, you can assume root
privileges and use chroot normally.
> The key question for us remains what value the abstraction can have for
> Isar, today and on the long run. Switching builders is not a daily
> business so far. Isar needs one, and that needs to plug well into the
> image creation workflow. If interacting with mdbp can be simpler than
> talking to sbuild directly, that would be a clear gain. If it's rather
> more complex under the Isar constraints while we do not have the
> compelling use case for switching builders, it is likely not the best
> way forward.
I think the value is not to be gained by you, but by Isar users. You
already found your preferred builder and it is easy to stick to that.
Others may have set up a different one already. The value gained from
the abstraction is to be able to seamlessly use the builder of the
user's choice.
Another aspect is the need for avoiding nesting of virtualization
technology. When building inside kas-isar, you cannot use user
namespaces (because secure docker containers cannot). When building on a
plain Debian system, you want to use user namespaces to be able to build
unprivileged. At present, the only builder that allows switching between
user namespaces and no user namespaces is sbuild. mdbp also gives that
flexibility by allowing you to seamlessly swap builders.
So if sbuild suits your need, then go for it rather than reinventing the
wheel.
I caution that using sbuild in a reproducible way is hard. People do
change its defaults to their needs. You need to pass a lot of options to
sbuild to avoid using user's defaults. That was one of the lessons I
learned the hard way when writing mdbp.
Helmut
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 13:52 ` Helmut Grohne
2021-07-08 17:16 ` Jan Kiszka
@ 2021-07-09 15:46 ` Cedric Hombourger
2021-07-09 16:12 ` Cedric Hombourger
1 sibling, 1 reply; 13+ messages in thread
From: Cedric Hombourger @ 2021-07-09 15:46 UTC (permalink / raw)
To: Helmut Grohne, Jan Kiszka; +Cc: isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 7/8/2021 3:52 PM, Helmut Grohne wrote:
> On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
>> On 08.07.21 11:07, Cedric Hombourger wrote:
>> ...
>> Longterm, there is also the desire to include support for DPKG_ROOT as
>> chroot-less way of building packages, faster when doing it cross and
>> also without special permissions (e.g. for qemu-user). But that requires
>> per-package support from Debian upstream. Discussions only started, in
>> particular with Helmuth (added to CC).
> You appear to be confusing some aspects here. DPKG_ROOT is not relevant
> to building packages. It is only relevant to installing them (which may
> be relevant here for creating filesystem images).
>
> Building packages without chroot in a reproducible way seems next to
> impossible to me. Even when you use user namespaces, chroot does not go
> away. It merely becomes unprivileged. Is that what you mean here?
>
>>> Proposal
>>>
>>> We may want to use unshare(1) to create a mount namespace where we
>>> will create our bind mounts,
>>> chroot into the buildchroot and run the specified command/script
> Are you aware that sbuild directly supports this use case? It has a
> --mode argument and one of its values is "unshare". In that case, you
> supply a tarball containing the chroot and it'll perform an unprivileged
> build inside an unshared chroot.
I was not and that's very promising. I am now modifying the PoC code to
use it.
Did I read correctly that we can tell sbuild to use an existing
directory for its chroot when using the "unshare" mode? I am asking
because that's failing for me (my host is on Debian/testing). Here's the
error
copy() failed: Is a directory
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
and here's the full command I ran (with the key args being -c
<existing-chroot-dir> --chroot-mode=unshare):
+ sbuild -c
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
--chroot-mode=unshare -d industrial-os --no-apt-update -v
--pre-build-commands= mkdir -p
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
; mount --bind
/home/chombourger/unshare_sbuild/experimental/mel-apt
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
; mount --bind
'/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/deploy/isar-apt/industrial-os-amd64/apt/industrial-os'
'/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/isar-apt'
; mount --bind
'/home/chombourger/unshare_sbuild/experimental/build-ipc/downloads'
'/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/downloads'
; mount -t proc none
'/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/proc'
; mount --rbind /sys
'/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/sys'
; mount -t tmpfs -o rw,nosuid,nodev,seclabel none
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/shm
; mount -o bind /dev/pts
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/pts
; mkdir -p
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
; mount --bind
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
--build-path=/home/builder/base-files/base-files-10.3+deb10u10 -D
and it produced the following output:
dh clean
dh: warning: Compatibility levels before 10 are deprecated (level 9
in use)
dh_clean
dh_clean: warning: Compatibility levels before 10 are deprecated
(level 9 in use)
dpkg-source: info: using source format '3.0 (native)'
dpkg-source: info: building base-files in base-files_2.4+ind3.tar.xz
dpkg-source: info: building base-files in base-files_2.4+ind3.dsc
Selected distribution industrial-os
Selected chroot
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
D: Setting ABORT=undef
D: Setting
Job=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
D: Setting Build Dir=
D: Setting Max Lock Trys=120
D: Setting Lock Interval=5
D: Setting Pkg Status=pending
D: Setting Pkg Status Trigger=undef
D: Setting Pkg Start Time=0
D: Setting Pkg End Time=0
D: Setting Pkg Fail Stage=init
D: Setting Build Start Time=0
D: Setting Build End Time=0
D: Setting Install Start Time=0
D: Setting Install End Time=0
D: Setting This Time=0
D: Setting This Space=0
D: Setting Sub Task=initialisation
D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
D: Setting Session ID=
D: Setting Chroot ID=/
D: Setting Defaults=HASH(0x55a7fc2a7e48)
D: Setting Split=1
D: Setting Split=0
D: Setting Host=Sbuild::ChrootRoot=HASH(0x55a7fc2a8010)
D: Setting Priority=0
D: Setting Location=/
D: Setting Session Purged=0
D: Setting Session=undef
D: Setting Dependency Resolver=undef
D: Setting Log File=undef
D: Setting Log Stream=undef
D: Setting Summary Stats=HASH(0x55a7fc283c70)
D: Setting dpkg-buildpackage pid=undef
D: Setting Dpkg Version=undef
D: Setting DSC:
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
D: Setting
DSC=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
D: Setting Source
Dir=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
D: Setting DSC Base=base-files_2.4+ind3.dsc
D: DSC =
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
D: Source Dir =
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
D: DSC Base = base-files_2.4+ind3.dsc
D: Setting package version:
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
D: Parsing
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
D: Setting Package=base-files
D: Setting Version=1:2.4+ind3
D: Setting Package_Version=base-files_1:2.4+ind3
D: Setting Package_OVersion=base-files_1:2.4+ind3
D: Setting Package_OSVersion=base-files_2.4+ind3
D: Setting Package_SVersion=base-files_2.4+ind3
D: Setting OVersion=1:2.4+ind3
D: Setting OSVersion=2.4+ind3
D: Setting SVersion=2.4+ind3
D: Setting VersionEpoch=1
D: Setting VersionUpstream=2.4+ind3
D: Setting VersionDebian=
D: Setting DSC File=base-files_2.4+ind3.dsc
D: Setting DSC Dir=base-files-2.4+ind3
D: Package = base-files
D: Version = 1:2.4+ind3
D: Package_Version = base-files_1:2.4+ind3
D: Package_OVersion = base-files_1:2.4+ind3
D: Package_OSVersion = base-files_2.4+ind3
D: Package_SVersion = base-files_2.4+ind3
D: OVersion = 1:2.4+ind3
D: OSVersion = 2.4+ind3
D: SVersion = 2.4+ind3
D: VersionEpoch = 1
D: VersionUpstream = 2.4+ind3
D: VersionDebian =
D: DSC File = base-files_2.4+ind3.dsc
D: DSC Dir = base-files-2.4+ind3
D: Setting Pkg Status Trigger=CODE(0x55a7fc2300c8)
D: Setting Pkg Status=building
D: Setting Pkg Start Time=1625844658
D: Setting Pkg End Time=1625844658
D: Setting Host Arch=amd64
D: Setting Build Arch=amd64
D: Setting Build Profiles=
D: Setting Build Type=binary
D: Setting FILTER_PREFIX=__SBUILD_FILTER_1412690:
D: Setting COLOUR_PREFIX=__SBUILD_COLOUR_1412690:
D: Setting Log
File=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3_amd64-2021-07-09T15:30:58Z.build
D: Setting Log Stream=GLOB(0x55a7fc28c788)
sbuild (Debian sbuild) 0.81.2 (31 January 2021) on build.local
+==============================================================================+
| base-files 1:2.4+ind3 (amd64) Fri, 09 Jul 2021
15:30:58 +0000 |
+==============================================================================+
Package: base-files
Version: 1:2.4+ind3
Source Version: 1:2.4+ind3
Distribution: industrial-os
Machine Architecture: amd64
Host Architecture: amd64
Build Architecture: amd64
Build Type: binary
D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
D: Setting Chroots=HASH(0x55a7fc28ff88)
I: No tarballs found in /home/builder/.cache/sbuild
D: Setting Chroots=HASH(0x55a7fc298bd0)
D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
D: Setting Session ID=
D: Setting Chroot
ID=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
D: Setting Defaults=HASH(0x55a7fc362e08)
D: Setting Chroots=Sbuild::ChrootInfoUnshare=HASH(0x55a7fc2880a0)
D: Setting Uid Gid Map=ARRAY(0x55a7fbdf7608)
running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
= $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
{close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
receive EOF";0 == system "newuidmap $ppid 0 1000 1 1 100000 1" or
die "newuidmap failed: $!";0 == system "newgidmap $ppid 0 1000 1 1
100000 1" or die "newgidmap failed: $!";exit 0;}0 == syscall
&SYS_unshare, 268435456 or die "unshare() failed: $!";close
$wfh;$cpid == waitpid $cpid, 0 or die "waitpid() failed: $!";if ($?
!= 0) {die "child had a non-zero exit status: $?";}0 == syscall
&SYS_setgid, 0 or die "setgid failed: $!";0 == syscall &SYS_setuid,
0 or die "setuid failed: $!";0 == syscall &SYS_setgroups, 0, 0 or
die "setgroups failed: $!";exec { $ARGV[0] } @ARGV or die "exec()
failed: $!"; chown 1:1 /tmp/tmp.sbuild.l7HJK58Vy2
Unpacking
/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs
to /tmp/tmp.sbuild.l7HJK58Vy2...
running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
= $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
{close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
receive EOF";0 == system "newuidmap $ppid 0 100000 65536" or die
"newuidmap failed: $!";0 == system "newgidmap $ppid 0 100000 65536"
or die "newgidmap failed: $!";exit 0;}0 == syscall &SYS_unshare,
268435456 or die "unshare() failed: $!";close $wfh;$cpid == waitpid
$cpid, 0 or die "waitpid() failed: $!";if ($? != 0) {die "child had
a non-zero exit status: $?";}0 == syscall &SYS_setgid, 0 or die
"setgid failed: $!";0 == syscall &SYS_setuid, 0 or die "setuid
failed: $!";0 == syscall &SYS_setgroups, 0, 0 or die "setgroups
failed: $!";exec { $ARGV[0] } @ARGV or die "exec() failed: $!"; tar
--exclude=./dev/urandom --exclude=./dev/random --exclude=./dev/full
--exclude=./dev/null --exclude=./dev/console --exclude=./dev/zero
--exclude=./dev/tty --exclude=./dev/ptmx --directory
/tmp/tmp.sbuild.l7HJK58Vy2 --extract
copy() failed: Is a directory
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
D: Error run_chroot_session(): Error creating chroot session:
skipping base-filesD: Setting Session=undef
D: Error run_chroot(): Error creating chroot session: skipping
base-filesE: Error creating chroot session: skipping base-files
D: Setting Pkg Status=failed
D: Setting Pkg Fail Stage=create-session
The extra bind-mounts would be needed for the following apt source to be
work:
$ cat
tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/etc/apt/sources.list.d/isar-apt.list
deb [trusted=yes] file:///isar-apt mel main
>
>>> The immediate benefit of this approach is that all mounts
>>> automatically disappear as the supplied
>>> command exits (whether it aborts prematurely because of an error or
>>> normally on completion).
>>>
>>> Another nice benefit is that bind mounts we created within this
>>> namespace are not (directly) visible
>>> from the parent namespace
>>>
>>> However, we found that running scripts within an unshare environment
>>> may not be as easy as
>>> chroot. We would welcome feedback on the code snippets provided
>>> below if you happen to have
>>> some better ideas.
> All of this applies to sbuild --mode=unshare as well except that it
> makes running scripts from hooks simple.
it certainly would!
>
>> I suspect Helmuth can tell us if that would take us on a fragile path
>> from Debian perspective. Isar-internal implementation details we could
>> likely sort out, but if that approach has architectural limits /wrt what
>> Debian packages expect/require, it might be the wrong direction.
> Reimplementing this functionality seems like a waste of time to me. If
> we ignore that for a moment, we notice that there are already ~10
> implementations and updating them all is painful. Therefore, we can
> conclude that changes to the build environment are rare and your
> reimplementation likely is maintainable with limited effort.
>
>>> def isar_user_spec():
>>> import os
>>> return '%d:%d' % (os.getuid(), os.getgid())
>>>
>>> ISAR_USER_SPEC = "${@ isar_user_spec() }"
>>> ISAR_UNSHARE_CMD = "sudo unshare --pid --fork --ipc --mount sh -ex"
>>> ISAR_CHROOT_SHELL = "sh -ex"
>>> ISAR_CHROOT_ROOT = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>> ISAR_CHROOT_USER = "chroot --userspec='${ISAR_USER_SPEC}'
>>> ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>
>>> # Would be similar to buildchroot_do_mounts but will happen in a
>>> separate mount namespace
>>> BUILDCHROOT_DO_MOUNTS =
>>> " \
>>> mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>>> '${BUILDCHROOT_DIR}/isar-apt' ; \
>>> mount --bind '${DL_DIR}'
>>> '${BUILDCHROOT_DIR}/downloads' ; \
>>> mount -t proc none
>>> '${BUILDCHROOT_DIR}/proc' ; \
>>> mount --rbind /sys
>>> '${BUILDCHROOT_DIR}/sys' ; \
>>> mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>>> ${BUILDCHROOT_DIR}/dev/shm ; \
>>> mount -o bind /dev/pts
>>> ${BUILDCHROOT_DIR}/dev/pts \
>>> "
>>>
>>> # Would be similar to dpkg_do_mounts but will happen in a separate
>>> mount namespace
>>> DPKG_DO_MOUNTS = " \
>>> ${BUILDCHROOT_DO_MOUNTS} ; \
>>> mkdir -p ${BUILDROOT} ; \
>>> mount --bind ${WORKDIR} ${BUILDROOT} \
>>> "
>>>
>>> # Build package from sources using build script
>>> _runbuild() {
>>> export arch=${1}
>>>
>>> E="${@ isar_export_proxies(d)}"
>>> ( cat <<" UNSHARE"
>>> ${DPKG_DO_MOUNTS}
>>> ( cat <<" SCRIPT"
>>> export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>>> export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>>> export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>> /isar/build.sh ${PP}/${PPS} ${arch}
>>> SCRIPT
>>> ) | ${ISAR_CHROOT_USER}
>>> UNSHARE
>>> ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>>> }
>>>
>>> dpkg_runbuild() {
>>> ( _runbuild ${PACKAGE_ARCH} )
>>> }
>>>
>>> PS: I am not very happy with the need to feed the script to execute
>>> under unshare
>>> via stdin, if there are better ways, we would be happy to
>>> consider them!
> When I started talking to Jan, I proposed adding an abstraction layer
> for package building. Work on that layer has now progressed under the
> name "mdbp" and source is available at
> https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
> Raphael Hertzog on unifying the API with debusine. Let us for a moment
> consider the implications of using mdbp here.
Thanks for the pointer. Yet another thing I probably want to look at
>
> * Much of the complexity would go away. What you are left with is
> writing a json file describing how you want your package built. What
> gets a little more difficult is getting the isar-apt repository past
> mdbp. Likely that would require a temporary http server.
> * mdbp is not another builder, but an adapter to existing ones. It can
> perform your builds using an existing sbuild or pbuilder
> installation. If you want more isolation, maybe using debspawn
> (backed by systemd-nspawn) is for you?
> * mdbp also provides a stateless backend that uses mmdebstrap. This
> backend performs the build in a user namespace.
> * If you decide that you prefer building in docker, we can add a
> backend for e.g. debocker or something else.
>
> I'm not sure what you'd be missing by using mdbp here, but one thing
> you'd certainly miss is quite a bit of complex code.
>
> If you want to consider this route, read the schema first:
> https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json
>
> Helmut
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-09 15:46 ` Cedric Hombourger
@ 2021-07-09 16:12 ` Cedric Hombourger
0 siblings, 0 replies; 13+ messages in thread
From: Cedric Hombourger @ 2021-07-09 16:12 UTC (permalink / raw)
To: Helmut Grohne, Jan Kiszka; +Cc: isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 7/9/2021 5:46 PM, Cedric Hombourger wrote:
>
> On 7/8/2021 3:52 PM, Helmut Grohne wrote:
>> On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
>>> On 08.07.21 11:07, Cedric Hombourger wrote:
>>> ...
>>> Longterm, there is also the desire to include support for DPKG_ROOT as
>>> chroot-less way of building packages, faster when doing it cross and
>>> also without special permissions (e.g. for qemu-user). But that
>>> requires
>>> per-package support from Debian upstream. Discussions only started, in
>>> particular with Helmuth (added to CC).
>> You appear to be confusing some aspects here. DPKG_ROOT is not relevant
>> to building packages. It is only relevant to installing them (which may
>> be relevant here for creating filesystem images).
>>
>> Building packages without chroot in a reproducible way seems next to
>> impossible to me. Even when you use user namespaces, chroot does not go
>> away. It merely becomes unprivileged. Is that what you mean here?
>>
>>>> Proposal
>>>>
>>>> We may want to use unshare(1) to create a mount namespace where we
>>>> will create our bind mounts,
>>>> chroot into the buildchroot and run the specified command/script
>> Are you aware that sbuild directly supports this use case? It has a
>> --mode argument and one of its values is "unshare". In that case, you
>> supply a tarball containing the chroot and it'll perform an unprivileged
>> build inside an unshared chroot.
>
> I was not and that's very promising. I am now modifying the PoC code
> to use it.
>
> Did I read correctly that we can tell sbuild to use an existing
> directory for its chroot when using the "unshare" mode? I am asking
> because that's failing for me (my host is on Debian/testing). Here's
> the error
looks like --chroot=<dir> isn't supported yet for "unshare" as found in
the sbuild code:
# FIXME: support directory chroots
#if (-d $path) {
# if ($file eq $chroot) {
# $tarball = $path;
# last;
# }
#} else {
>
> copy() failed: Is a directory
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
>
> and here's the full command I ran (with the key args being -c
> <existing-chroot-dir> --chroot-mode=unshare):
>
> + sbuild -c
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
> --chroot-mode=unshare -d industrial-os --no-apt-update -v
> --pre-build-commands= mkdir -p
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
> ; mount --bind
> /home/chombourger/unshare_sbuild/experimental/mel-apt
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
> ; mount --bind
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/deploy/isar-apt/industrial-os-amd64/apt/industrial-os'
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/isar-apt'
> ; mount --bind
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/downloads'
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/downloads'
> ; mount -t proc none
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/proc'
> ; mount --rbind /sys
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/sys'
> ; mount -t tmpfs -o rw,nosuid,nodev,seclabel none
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/shm
> ; mount -o bind /dev/pts
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/pts
> ; mkdir -p
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
> ; mount --bind
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
> --build-path=/home/builder/base-files/base-files-10.3+deb10u10 -D
>
> and it produced the following output:
>
> dh clean
> dh: warning: Compatibility levels before 10 are deprecated (level 9
> in use)
> dh_clean
> dh_clean: warning: Compatibility levels before 10 are deprecated
> (level 9 in use)
> dpkg-source: info: using source format '3.0 (native)'
> dpkg-source: info: building base-files in base-files_2.4+ind3.tar.xz
> dpkg-source: info: building base-files in base-files_2.4+ind3.dsc
> Selected distribution industrial-os
> Selected chroot
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
> D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
> D: Setting ABORT=undef
> D: Setting
> Job=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
> D: Setting Build Dir=
> D: Setting Max Lock Trys=120
> D: Setting Lock Interval=5
> D: Setting Pkg Status=pending
> D: Setting Pkg Status Trigger=undef
> D: Setting Pkg Start Time=0
> D: Setting Pkg End Time=0
> D: Setting Pkg Fail Stage=init
> D: Setting Build Start Time=0
> D: Setting Build End Time=0
> D: Setting Install Start Time=0
> D: Setting Install End Time=0
> D: Setting This Time=0
> D: Setting This Space=0
> D: Setting Sub Task=initialisation
> D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
> D: Setting Session ID=
> D: Setting Chroot ID=/
> D: Setting Defaults=HASH(0x55a7fc2a7e48)
> D: Setting Split=1
> D: Setting Split=0
> D: Setting Host=Sbuild::ChrootRoot=HASH(0x55a7fc2a8010)
> D: Setting Priority=0
> D: Setting Location=/
> D: Setting Session Purged=0
> D: Setting Session=undef
> D: Setting Dependency Resolver=undef
> D: Setting Log File=undef
> D: Setting Log Stream=undef
> D: Setting Summary Stats=HASH(0x55a7fc283c70)
> D: Setting dpkg-buildpackage pid=undef
> D: Setting Dpkg Version=undef
> D: Setting DSC:
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
> D: Setting
> DSC=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
> D: Setting Source
> Dir=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
> D: Setting DSC Base=base-files_2.4+ind3.dsc
> D: DSC =
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
> D: Source Dir =
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
> D: DSC Base = base-files_2.4+ind3.dsc
> D: Setting package version:
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
> D: Parsing
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
> D: Setting Package=base-files
> D: Setting Version=1:2.4+ind3
> D: Setting Package_Version=base-files_1:2.4+ind3
> D: Setting Package_OVersion=base-files_1:2.4+ind3
> D: Setting Package_OSVersion=base-files_2.4+ind3
> D: Setting Package_SVersion=base-files_2.4+ind3
> D: Setting OVersion=1:2.4+ind3
> D: Setting OSVersion=2.4+ind3
> D: Setting SVersion=2.4+ind3
> D: Setting VersionEpoch=1
> D: Setting VersionUpstream=2.4+ind3
> D: Setting VersionDebian=
> D: Setting DSC File=base-files_2.4+ind3.dsc
> D: Setting DSC Dir=base-files-2.4+ind3
> D: Package = base-files
> D: Version = 1:2.4+ind3
> D: Package_Version = base-files_1:2.4+ind3
> D: Package_OVersion = base-files_1:2.4+ind3
> D: Package_OSVersion = base-files_2.4+ind3
> D: Package_SVersion = base-files_2.4+ind3
> D: OVersion = 1:2.4+ind3
> D: OSVersion = 2.4+ind3
> D: SVersion = 2.4+ind3
> D: VersionEpoch = 1
> D: VersionUpstream = 2.4+ind3
> D: VersionDebian =
> D: DSC File = base-files_2.4+ind3.dsc
> D: DSC Dir = base-files-2.4+ind3
> D: Setting Pkg Status Trigger=CODE(0x55a7fc2300c8)
> D: Setting Pkg Status=building
> D: Setting Pkg Start Time=1625844658
> D: Setting Pkg End Time=1625844658
> D: Setting Host Arch=amd64
> D: Setting Build Arch=amd64
> D: Setting Build Profiles=
> D: Setting Build Type=binary
> D: Setting FILTER_PREFIX=__SBUILD_FILTER_1412690:
> D: Setting COLOUR_PREFIX=__SBUILD_COLOUR_1412690:
> D: Setting Log
> File=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3_amd64-2021-07-09T15:30:58Z.build
> D: Setting Log Stream=GLOB(0x55a7fc28c788)
> sbuild (Debian sbuild) 0.81.2 (31 January 2021) on build.local
>
> +==============================================================================+
> | base-files 1:2.4+ind3 (amd64) Fri, 09 Jul 2021
> 15:30:58 +0000 |
> +==============================================================================+
>
> Package: base-files
> Version: 1:2.4+ind3
> Source Version: 1:2.4+ind3
> Distribution: industrial-os
> Machine Architecture: amd64
> Host Architecture: amd64
> Build Architecture: amd64
> Build Type: binary
>
> D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
> D: Setting Chroots=HASH(0x55a7fc28ff88)
> I: No tarballs found in /home/builder/.cache/sbuild
> D: Setting Chroots=HASH(0x55a7fc298bd0)
> D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
> D: Setting Session ID=
> D: Setting Chroot
> ID=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
> D: Setting Defaults=HASH(0x55a7fc362e08)
> D: Setting Chroots=Sbuild::ChrootInfoUnshare=HASH(0x55a7fc2880a0)
> D: Setting Uid Gid Map=ARRAY(0x55a7fbdf7608)
> running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
> = $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
> {close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
> receive EOF";0 == system "newuidmap $ppid 0 1000 1 1 100000 1" or
> die "newuidmap failed: $!";0 == system "newgidmap $ppid 0 1000 1 1
> 100000 1" or die "newgidmap failed: $!";exit 0;}0 == syscall
> &SYS_unshare, 268435456 or die "unshare() failed: $!";close
> $wfh;$cpid == waitpid $cpid, 0 or die "waitpid() failed: $!";if ($?
> != 0) {die "child had a non-zero exit status: $?";}0 == syscall
> &SYS_setgid, 0 or die "setgid failed: $!";0 == syscall &SYS_setuid,
> 0 or die "setuid failed: $!";0 == syscall &SYS_setgroups, 0, 0 or
> die "setgroups failed: $!";exec { $ARGV[0] } @ARGV or die "exec()
> failed: $!"; chown 1:1 /tmp/tmp.sbuild.l7HJK58Vy2
> Unpacking
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs
> to /tmp/tmp.sbuild.l7HJK58Vy2...
> running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
> = $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
> {close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
> receive EOF";0 == system "newuidmap $ppid 0 100000 65536" or die
> "newuidmap failed: $!";0 == system "newgidmap $ppid 0 100000 65536"
> or die "newgidmap failed: $!";exit 0;}0 == syscall &SYS_unshare,
> 268435456 or die "unshare() failed: $!";close $wfh;$cpid == waitpid
> $cpid, 0 or die "waitpid() failed: $!";if ($? != 0) {die "child had
> a non-zero exit status: $?";}0 == syscall &SYS_setgid, 0 or die
> "setgid failed: $!";0 == syscall &SYS_setuid, 0 or die "setuid
> failed: $!";0 == syscall &SYS_setgroups, 0, 0 or die "setgroups
> failed: $!";exec { $ARGV[0] } @ARGV or die "exec() failed: $!"; tar
> --exclude=./dev/urandom --exclude=./dev/random --exclude=./dev/full
> --exclude=./dev/null --exclude=./dev/console --exclude=./dev/zero
> --exclude=./dev/tty --exclude=./dev/ptmx --directory
> /tmp/tmp.sbuild.l7HJK58Vy2 --extract
> copy() failed: Is a directory
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
> D: Error run_chroot_session(): Error creating chroot session:
> skipping base-filesD: Setting Session=undef
> D: Error run_chroot(): Error creating chroot session: skipping
> base-filesE: Error creating chroot session: skipping base-files
> D: Setting Pkg Status=failed
> D: Setting Pkg Fail Stage=create-session
>
>
> The extra bind-mounts would be needed for the following apt source to
> be work:
>
> $ cat
> tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/etc/apt/sources.list.d/isar-apt.list
> deb [trusted=yes] file:///isar-apt mel main
>
>>
>>>> The immediate benefit of this approach is that all mounts
>>>> automatically disappear as the supplied
>>>> command exits (whether it aborts prematurely because of an
>>>> error or
>>>> normally on completion).
>>>>
>>>> Another nice benefit is that bind mounts we created within this
>>>> namespace are not (directly) visible
>>>> from the parent namespace
>>>>
>>>> However, we found that running scripts within an unshare
>>>> environment
>>>> may not be as easy as
>>>> chroot. We would welcome feedback on the code snippets provided
>>>> below if you happen to have
>>>> some better ideas.
>> All of this applies to sbuild --mode=unshare as well except that it
>> makes running scripts from hooks simple.
> it certainly would!
>>
>>> I suspect Helmuth can tell us if that would take us on a fragile path
>>> from Debian perspective. Isar-internal implementation details we could
>>> likely sort out, but if that approach has architectural limits /wrt
>>> what
>>> Debian packages expect/require, it might be the wrong direction.
>> Reimplementing this functionality seems like a waste of time to me. If
>> we ignore that for a moment, we notice that there are already ~10
>> implementations and updating them all is painful. Therefore, we can
>> conclude that changes to the build environment are rare and your
>> reimplementation likely is maintainable with limited effort.
>>
>>>> def isar_user_spec():
>>>> import os
>>>> return '%d:%d' % (os.getuid(), os.getgid())
>>>>
>>>> ISAR_USER_SPEC = "${@ isar_user_spec() }"
>>>> ISAR_UNSHARE_CMD = "sudo unshare --pid --fork --ipc --mount sh
>>>> -ex"
>>>> ISAR_CHROOT_SHELL = "sh -ex"
>>>> ISAR_CHROOT_ROOT = "chroot ${BUILDCHROOT_DIR}
>>>> ${ISAR_CHROOT_SHELL}"
>>>> ISAR_CHROOT_USER = "chroot --userspec='${ISAR_USER_SPEC}'
>>>> ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>>
>>>> # Would be similar to buildchroot_do_mounts but will happen in a
>>>> separate mount namespace
>>>> BUILDCHROOT_DO_MOUNTS =
>>>> " \
>>>> mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>>>> '${BUILDCHROOT_DIR}/isar-apt' ; \
>>>> mount --bind '${DL_DIR}'
>>>> '${BUILDCHROOT_DIR}/downloads' ; \
>>>> mount -t proc none
>>>> '${BUILDCHROOT_DIR}/proc' ; \
>>>> mount --rbind /sys
>>>> '${BUILDCHROOT_DIR}/sys' ; \
>>>> mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>>>> ${BUILDCHROOT_DIR}/dev/shm ; \
>>>> mount -o bind /dev/pts
>>>> ${BUILDCHROOT_DIR}/dev/pts \
>>>> "
>>>>
>>>> # Would be similar to dpkg_do_mounts but will happen in a separate
>>>> mount namespace
>>>> DPKG_DO_MOUNTS = " \
>>>> ${BUILDCHROOT_DO_MOUNTS} ; \
>>>> mkdir -p ${BUILDROOT} ; \
>>>> mount --bind ${WORKDIR} ${BUILDROOT} \
>>>> "
>>>>
>>>> # Build package from sources using build script
>>>> _runbuild() {
>>>> export arch=${1}
>>>>
>>>> E="${@ isar_export_proxies(d)}"
>>>> ( cat <<" UNSHARE"
>>>> ${DPKG_DO_MOUNTS}
>>>> ( cat <<" SCRIPT"
>>>> export
>>>> DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>>>> export
>>>> DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>>>> export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>> /isar/build.sh ${PP}/${PPS} ${arch}
>>>> SCRIPT
>>>> ) | ${ISAR_CHROOT_USER}
>>>> UNSHARE
>>>> ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>>>> }
>>>>
>>>> dpkg_runbuild() {
>>>> ( _runbuild ${PACKAGE_ARCH} )
>>>> }
>>>>
>>>> PS: I am not very happy with the need to feed the script to
>>>> execute
>>>> under unshare
>>>> via stdin, if there are better ways, we would be happy to
>>>> consider them!
>> When I started talking to Jan, I proposed adding an abstraction layer
>> for package building. Work on that layer has now progressed under the
>> name "mdbp" and source is available at
>> https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
>> Raphael Hertzog on unifying the API with debusine. Let us for a moment
>> consider the implications of using mdbp here.
> Thanks for the pointer. Yet another thing I probably want to look at
>>
>> * Much of the complexity would go away. What you are left with is
>> writing a json file describing how you want your package built. What
>> gets a little more difficult is getting the isar-apt repository past
>> mdbp. Likely that would require a temporary http server.
>> * mdbp is not another builder, but an adapter to existing ones. It can
>> perform your builds using an existing sbuild or pbuilder
>> installation. If you want more isolation, maybe using debspawn
>> (backed by systemd-nspawn) is for you?
>> * mdbp also provides a stateless backend that uses mmdebstrap. This
>> backend performs the build in a user namespace.
>> * If you decide that you prefer building in docker, we can add a
>> backend for e.g. debocker or something else.
>>
>> I'm not sure what you'd be missing by using mdbp here, but one thing
>> you'd certainly miss is quite a bit of complex code.
>>
>> If you want to consider this route, read the schema first:
>> https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json
>>
>>
>> Helmut
>>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 19:34 ` Helmut Grohne
@ 2021-07-12 8:25 ` Jan Kiszka
2021-07-12 10:54 ` Helmut Grohne
0 siblings, 1 reply; 13+ messages in thread
From: Jan Kiszka @ 2021-07-12 8:25 UTC (permalink / raw)
To: Helmut Grohne
Cc: Cedric Hombourger, isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 08.07.21 21:34, Helmut Grohne wrote:
> Hi Jan,
>
> On Thu, Jul 08, 2021 at 07:16:05PM +0200, Jan Kiszka wrote:
>> Yeah, not cleanly separated: We need DPKG_ROOT to install build
>> dependencies unprivileged into a build environment, and then we need
>> something ideally unprivileged - e.g. namespaces - to run the build
>> inside that environment.
>
> Still wrong. DPKG_ROOT is not meant to install build dependencies. I
> doubt that it covers sufficient packages to do so any time soon. I also
> don't see any point in doing so when you can just chroot (unprivileged
> via user namespaces) into it and install them there.
>
> The reason to want DPKG_ROOT is to prepare the root filesystem for your
> embedded board. If that's not what you want to do, DPKG_ROOT is not your
> tool.
We want it for both the board and the build env. We have the same issues
to solve there, means installing packages in unprivileged manner, either
for target arch or the builder arch (cross and native builds need to be
supported). So I see DPKG_ROOT as a building block for both. In
addition, it would overcome qemu-user, which is speed gain.
But, as you mentioned, that it all long-term, nothing to achieve soon
with a sufficiently large set of packages. It's still important to
develop a vision of where we want to go once possible.
>
>> We don't want to reimplement if key requirements can be fulfilled with
>> Debian's own tools. One of them is having control over how build
>> dependencies are installed and from where (use cases: isar-apt, offline
>> builds). Another one would be the ability to patch an original package
>> after unpacking its sources.
>
> I think the goal should be making Debian's tools fulfil your key
> requirements. (Regardless of whether using an abstraction layer or not.)
>
> Can you elaborate how you want to control your build dependencies? It
> would seem to me that you could just patch debian/control and be done,
> no? What else do you need to control? Do you want to change the
> resolver?
Yes, we prepare (ad-hoc debianization) or patch or just use
debian/control for that. Packages should then be taken from the
configured deb repos, including local isar-apt, and using a pre-defined
apt database (only updated once so that all builds use the same package
versions). Local caching of the fetched deb packages is used in
addition, both as optimization as well as a building block for offline
builds.
>
> The "from where" part seems relatively easy to me. All of the build
> tools support one way or another to supply extra repositories. All you
> need to do here is ensure that your extra packages from extra
> repositories have a higher version than those in your base distribution.
>
> Why does your build tool need to support patching after unpacking
> sources? I'm using that approach in rebootstrap and I consider it a
> design mistake. You can just extract your source package outside the
> builder, bump the version, apply your patches, create a new source
> package and build that. Creating actual source packages saves you from a
> lot of trouble.
One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g.
to inject patches before they are accepted upstream or to address issues
that are not compatible with upstream packaging. Recent example:
https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955
(some patches not even accepted upstream yet but needed).
This procedure should be automated and expressed with very few
statements. The current approaches we have are far from optimal, but the
vision remains that you just tell Isar the package, the patch, and then
it does whatever is best from Debian perspective. That may include the
mentioned source package patching upfront. That will have to go into our
recipes and classes.
>
>
>> Why can't that also support a local folder with a apt repo? Spawning
>> servers is surely doable but can become fun (unless using containers...)
>> when running multiple build in parallel.
>
> I've considered that, but support for this is lacking at this point.
> Ideally, you'd want to bind mount the local folder into the chroot. When
> using user namespaces your mapped root user may be underprivileged to
> see that local folder already. sbuild does not provide such means
> (unless you modify the underlying schroot, which requires root).
> pbuilder has --bindmounts (but doesn't use a user namespace). mmdebstrap
> can only sync directories, which means copying them.
>
>> If that is not going to be rebuilt for every package (cached baseline,
>> some dependency versions for all built packages in a run) and could
>> possibly also be reused for other things (running the imaging), it could
>> be a building block to replace our buildchroots. If not, we would have
>> to continue feeding it with separately bootstrapped chroots.
>
> Where does that "not rebuilt" requirement come from? Is that for
> performance reasons?
Sure. We are not building hundreds of packages, but it also not only
about building one or two. The other reason is the mentioned identical
versions in all buildchroots. Reproducing them must provide identical
content (where common) for a single image build run.
>
> If yes, please benchmark it. I've locally compared build times
> (sbuild/pbuilder/mmdebstrap) and the difference is small (<10 seconds).
> After all, extracting a pbuilder (or sbuild) base.tgz is not immediate
> either and mmdebstrap is quite fast.
>
>> Most of the Isar builds (complete images, not just packages) are already
>> happening inside containers (kas-isar). Thus we rather need to be
>> careful not requiring any nesting that could become complex - at best.
>> At the same time, we want Isar to continue work on a plain Debian system.
>
> You cannot use user namespaces in secure docker containers anyway, but
> then when you already are inside a container, you can assume root
> privileges and use chroot normally.
Currently, we have no secure container env. The container simply serves
as deployment tool of Isar's dependencies and builder-side
configurations so that you can run it on any distro, not only Debian.
And you can run it in CI identically to your host.
>
>> The key question for us remains what value the abstraction can have for
>> Isar, today and on the long run. Switching builders is not a daily
>> business so far. Isar needs one, and that needs to plug well into the
>> image creation workflow. If interacting with mdbp can be simpler than
>> talking to sbuild directly, that would be a clear gain. If it's rather
>> more complex under the Isar constraints while we do not have the
>> compelling use case for switching builders, it is likely not the best
>> way forward.
>
> I think the value is not to be gained by you, but by Isar users. You
> already found your preferred builder and it is easy to stick to that.
> Others may have set up a different one already. The value gained from
> the abstraction is to be able to seamlessly use the builder of the
> user's choice.
Isar users use Isar, they usually do not modify the core. And the goal
of Isar is to avoid that they have to but rather provide easy means to
configure things that must be configurable. If the builder would fall
under this, e.g. no one build would build all types of packages (hope
that is not the case, so far it wasn't), it would become an Isar
requirement to switching them without much effort.
>
> Another aspect is the need for avoiding nesting of virtualization
> technology. When building inside kas-isar, you cannot use user
> namespaces (because secure docker containers cannot). When building on a
> plain Debian system, you want to use user namespaces to be able to build
> unprivileged. At present, the only builder that allows switching between
> user namespaces and no user namespaces is sbuild. mdbp also gives that
> flexibility by allowing you to seamlessly swap builders.
>
> So if sbuild suits your need, then go for it rather than reinventing the
> wheel.
>
> I caution that using sbuild in a reproducible way is hard. People do
> change its defaults to their needs. You need to pass a lot of options to
> sbuild to avoid using user's defaults. That was one of the lessons I
> learned the hard way when writing mdbp.
>
That is a valid point we need to keep an eye on. If mdbp solved that
already, it would be one of the added values I mentioned that could make
it relevant for Isar, even if not requiring to switch builders.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-12 8:25 ` Jan Kiszka
@ 2021-07-12 10:54 ` Helmut Grohne
2021-07-12 11:47 ` Jan Kiszka
0 siblings, 1 reply; 13+ messages in thread
From: Helmut Grohne @ 2021-07-12 10:54 UTC (permalink / raw)
To: Jan Kiszka
Cc: Cedric Hombourger, isar-users, Baurzhan Ismagulov, MacDonald, Joe
Hi Jan,
On Mon, Jul 12, 2021 at 10:25:41AM +0200, Jan Kiszka wrote:
> On 08.07.21 21:34, Helmut Grohne wrote:
> > The reason to want DPKG_ROOT is to prepare the root filesystem for your
> > embedded board. If that's not what you want to do, DPKG_ROOT is not your
> > tool.
>
> We want it for both the board and the build env. We have the same issues
> to solve there, means installing packages in unprivileged manner, either
> for target arch or the builder arch (cross and native builds need to be
> supported). So I see DPKG_ROOT as a building block for both. In
> addition, it would overcome qemu-user, which is speed gain.
No. Let us for a moment assume that DPKG_ROOT would just work for all
packages in Debian (and that is a far stretch really). Then you'd be
able to install your Build-Depends into a directory without using
chroot. But how do you build your package then? You need to chroot the
dpkg-buildpackage call. So you are back to requiring chroot, at which
point you can just install your whole build environment using chroot.
DPKG_ROOT is the wrong tool for that job. The one thing that it fixes is
requiring qemu for the embedded filesystem image. Cross builds already
work entirely without qemu (including package installation).
> Yes, we prepare (ad-hoc debianization) or patch or just use
> debian/control for that. Packages should then be taken from the
> configured deb repos, including local isar-apt, and using a pre-defined
> apt database (only updated once so that all builds use the same package
> versions). Local caching of the fetched deb packages is used in
> addition, both as optimization as well as a building block for offline
> builds.
I'm sorry, but none of this explains why you need to patch a package
after it is unpacked.
Before building starts, you can just download all the packages that are
to be patched, patch them and include the patched ones in isar-apt. At
that point, you can run without hooks.
> One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g.
> to inject patches before they are accepted upstream or to address issues
> that are not compatible with upstream packaging. Recent example:
> https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955
> (some patches not even accepted upstream yet but needed).
I do see why you need to patch packages. I do not see why that needs to
happen inside the build. Why not do it before building?
> This procedure should be automated and expressed with very few
> statements. The current approaches we have are far from optimal, but the
> vision remains that you just tell Isar the package, the patch, and then
> it does whatever is best from Debian perspective. That may include the
> mentioned source package patching upfront. That will have to go into our
> recipes and classes.
That sounds a lot like your requirement is a consequence of your current
implementation and not an actual requirement. I fully agree that such
patching needs to be simple to perform and (unfortunately) is a frequent
thing to do. Just the how we do it is sub-optimal both in Isar and
rebootstrap.
> Sure. We are not building hundreds of packages, but it also not only
> about building one or two. The other reason is the mentioned identical
> versions in all buildchroots. Reproducing them must provide identical
> content (where common) for a single image build run.
I think I already debunked the performance aspect. For reproducibility,
you do want a clean build to reproduce the artifacts of a previous one.
Reproducibility needs to work regardless of whether you cache the base
image. Indeed, the reproducible folks are working on reproducible
installations somewhat and the dpkg-root-demo explicitly verifies
reproducibility (given the same inputs, but you do cache them anyway for
offline builds).
So again, the requirement of caching an image seems to be a consequence
of your current implementation.
> Currently, we have no secure container env. The container simply serves
> as deployment tool of Isar's dependencies and builder-side
> configurations so that you can run it on any distro, not only Debian.
> And you can run it in CI identically to your host.
If your docker container is not setup securely, you can make it support
user namespaces and be done. :)
> Isar users use Isar, they usually do not modify the core. And the goal
> of Isar is to avoid that they have to but rather provide easy means to
> configure things that must be configurable. If the builder would fall
> under this, e.g. no one build would build all types of packages (hope
> that is not the case, so far it wasn't), it would become an Isar
> requirement to switching them without much effort.
How would I disagree with that? Of course providing users with the
flexibility to switch their builder turns out to become a requirement
once users rely on it. However, the "without much effort" part is the
thing that mdbp is supposed to provide. In theory, your builder becomes
a shell command with a sane default and your user can change that one
command to switch the builder, because the API differences are hidden
behind a common mdbp API.
> That is a valid point we need to keep an eye on. If mdbp solved that
> already, it would be one of the added values I mentioned that could make
> it relevant for Isar, even if not requiring to switch builders.
Great. Given that builder-flexibility seems to be a non-requirement for
you, I suggest that you use sbuild and look into the mdbp-sbuild
implementation to figure what kind of settings you need to fix to make
it reproducible. That's also a form of using it. ;)
Helmut
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-12 10:54 ` Helmut Grohne
@ 2021-07-12 11:47 ` Jan Kiszka
2021-07-12 12:29 ` Cedric Hombourger
0 siblings, 1 reply; 13+ messages in thread
From: Jan Kiszka @ 2021-07-12 11:47 UTC (permalink / raw)
To: Helmut Grohne
Cc: Cedric Hombourger, isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 12.07.21 12:54, Helmut Grohne wrote:
> Hi Jan,
>
> On Mon, Jul 12, 2021 at 10:25:41AM +0200, Jan Kiszka wrote:
>> On 08.07.21 21:34, Helmut Grohne wrote:
>>> The reason to want DPKG_ROOT is to prepare the root filesystem for your
>>> embedded board. If that's not what you want to do, DPKG_ROOT is not your
>>> tool.
>>
>> We want it for both the board and the build env. We have the same issues
>> to solve there, means installing packages in unprivileged manner, either
>> for target arch or the builder arch (cross and native builds need to be
>> supported). So I see DPKG_ROOT as a building block for both. In
>> addition, it would overcome qemu-user, which is speed gain.
>
> No. Let us for a moment assume that DPKG_ROOT would just work for all
> packages in Debian (and that is a far stretch really). Then you'd be
> able to install your Build-Depends into a directory without using
> chroot. But how do you build your package then? You need to chroot the
> dpkg-buildpackage call. So you are back to requiring chroot, at which
> point you can just install your whole build environment using chroot.
>
> DPKG_ROOT is the wrong tool for that job. The one thing that it fixes is
> requiring qemu for the embedded filesystem image. Cross builds already
> work entirely without qemu (including package installation).
>
We can't cross-build all packages, unfortunately. That's why we will
continue to require native installations for some packages'
buildchroots, just like we obviously do for the target rootfs. If you
can create and also augment (install deps) those without qemu, you gain
speed. Granted, those continue to require qemu for the actual build.
Which brings me back to still having to add namespace support to
binfmt_misc...
>> Yes, we prepare (ad-hoc debianization) or patch or just use
>> debian/control for that. Packages should then be taken from the
>> configured deb repos, including local isar-apt, and using a pre-defined
>> apt database (only updated once so that all builds use the same package
>> versions). Local caching of the fetched deb packages is used in
>> addition, both as optimization as well as a building block for offline
>> builds.
>
> I'm sorry, but none of this explains why you need to patch a package
> after it is unpacked.
>
> Before building starts, you can just download all the packages that are
> to be patched, patch them and include the patched ones in isar-apt. At
> that point, you can run without hooks.
Which implies fetching and unpacking them first. Yes, that might happen
in Isar without bothering the builder, and that might be a way to not
having to intercept its regular run. We don't do that right now, we just
kick off dpkg-buildpackage on the unpacked sources.
We also do that so far for packages that don't come as source packages
in the first place but as debianized repos or as ad-hoc debianized
sources (deb_debianize or local debian/ folder coming from an Isar
layer). Those would require an upfront source package creation as well,
something that dpkg-buildpackage currently does while building. Possibly
cleaner.
>
>> One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g.
>> to inject patches before they are accepted upstream or to address issues
>> that are not compatible with upstream packaging. Recent example:
>> https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955
>> (some patches not even accepted upstream yet but needed).
>
> I do see why you need to patch packages. I do not see why that needs to
> happen inside the build. Why not do it before building?
As long as it happens in the same Isar build run, that is fine, I agree.
>
>> This procedure should be automated and expressed with very few
>> statements. The current approaches we have are far from optimal, but the
>> vision remains that you just tell Isar the package, the patch, and then
>> it does whatever is best from Debian perspective. That may include the
>> mentioned source package patching upfront. That will have to go into our
>> recipes and classes.
>
> That sounds a lot like your requirement is a consequence of your current
> implementation and not an actual requirement. I fully agree that such
> patching needs to be simple to perform and (unfortunately) is a frequent
> thing to do. Just the how we do it is sub-optimal both in Isar and
> rebootstrap.
We are not bound to the current implementation, just the "user
interface" should not be needlessly changed. Means that the user
provides source URIs, artifacts etc. via bitbake recipes and gets a
patched and (re-)built package when building the related target.
>
>> Sure. We are not building hundreds of packages, but it also not only
>> about building one or two. The other reason is the mentioned identical
>> versions in all buildchroots. Reproducing them must provide identical
>> content (where common) for a single image build run.
>
> I think I already debunked the performance aspect. For reproducibility,
> you do want a clean build to reproduce the artifacts of a previous one.
> Reproducibility needs to work regardless of whether you cache the base
> image. Indeed, the reproducible folks are working on reproducible
> installations somewhat and the dpkg-root-demo explicitly verifies
> reproducibility (given the same inputs, but you do cache them anyway for
> offline builds).
>
> So again, the requirement of caching an image seems to be a consequence
> of your current implementation.
In parts but not generally, as you pointed out yourself ("given the same
inputs").
>
>> Currently, we have no secure container env. The container simply serves
>> as deployment tool of Isar's dependencies and builder-side
>> configurations so that you can run it on any distro, not only Debian.
>> And you can run it in CI identically to your host.
>
> If your docker container is not setup securely, you can make it support
> user namespaces and be done. :)
You seem to confuse containers and docker. Our vision is unprivileged
podman (or also docker-whatever, if that is still around by then) being
able to build complete images. Still a long way to go, though.
>
>> Isar users use Isar, they usually do not modify the core. And the goal
>> of Isar is to avoid that they have to but rather provide easy means to
>> configure things that must be configurable. If the builder would fall
>> under this, e.g. no one build would build all types of packages (hope
>> that is not the case, so far it wasn't), it would become an Isar
>> requirement to switching them without much effort.
>
> How would I disagree with that? Of course providing users with the
> flexibility to switch their builder turns out to become a requirement
> once users rely on it. However, the "without much effort" part is the
> thing that mdbp is supposed to provide. In theory, your builder becomes
> a shell command with a sane default and your user can change that one
> command to switch the builder, because the API differences are hidden
> behind a common mdbp API.
>
>> That is a valid point we need to keep an eye on. If mdbp solved that
>> already, it would be one of the added values I mentioned that could make
>> it relevant for Isar, even if not requiring to switch builders.
>
> Great. Given that builder-flexibility seems to be a non-requirement for
> you, I suggest that you use sbuild and look into the mdbp-sbuild
> implementation to figure what kind of settings you need to fix to make
> it reproducible. That's also a form of using it. ;)
>
Good! :)
Thanks,
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-12 11:47 ` Jan Kiszka
@ 2021-07-12 12:29 ` Cedric Hombourger
2021-07-12 14:35 ` Jan Kiszka
0 siblings, 1 reply; 13+ messages in thread
From: Cedric Hombourger @ 2021-07-12 12:29 UTC (permalink / raw)
To: Jan Kiszka, Helmut Grohne; +Cc: isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 7/12/2021 1:47 PM, Jan Kiszka wrote:
> On 12.07.21 12:54, Helmut Grohne wrote:
>> Hi Jan,
>>
>> On Mon, Jul 12, 2021 at 10:25:41AM +0200, Jan Kiszka wrote:
>>> On 08.07.21 21:34, Helmut Grohne wrote:
>>>> The reason to want DPKG_ROOT is to prepare the root filesystem for your
>>>> embedded board. If that's not what you want to do, DPKG_ROOT is not your
>>>> tool.
>>> We want it for both the board and the build env. We have the same issues
>>> to solve there, means installing packages in unprivileged manner, either
>>> for target arch or the builder arch (cross and native builds need to be
>>> supported). So I see DPKG_ROOT as a building block for both. In
>>> addition, it would overcome qemu-user, which is speed gain.
>> No. Let us for a moment assume that DPKG_ROOT would just work for all
>> packages in Debian (and that is a far stretch really). Then you'd be
>> able to install your Build-Depends into a directory without using
>> chroot. But how do you build your package then? You need to chroot the
>> dpkg-buildpackage call. So you are back to requiring chroot, at which
>> point you can just install your whole build environment using chroot.
>>
>> DPKG_ROOT is the wrong tool for that job. The one thing that it fixes is
>> requiring qemu for the embedded filesystem image. Cross builds already
>> work entirely without qemu (including package installation).
>>
> We can't cross-build all packages, unfortunately. That's why we will
> continue to require native installations for some packages'
> buildchroots, just like we obviously do for the target rootfs. If you
> can create and also augment (install deps) those without qemu, you gain
> speed. Granted, those continue to require qemu for the actual build.
> Which brings me back to still having to add namespace support to
> binfmt_misc...
there is that: https://patchwork.criu.org/series/3855/
I haven't checked if the author got any luck in getting his patchset
accepted
>
>>> Yes, we prepare (ad-hoc debianization) or patch or just use
>>> debian/control for that. Packages should then be taken from the
>>> configured deb repos, including local isar-apt, and using a pre-defined
>>> apt database (only updated once so that all builds use the same package
>>> versions). Local caching of the fetched deb packages is used in
>>> addition, both as optimization as well as a building block for offline
>>> builds.
>> I'm sorry, but none of this explains why you need to patch a package
>> after it is unpacked.
>>
>> Before building starts, you can just download all the packages that are
>> to be patched, patch them and include the patched ones in isar-apt. At
>> that point, you can run without hooks.
> Which implies fetching and unpacking them first. Yes, that might happen
> in Isar without bothering the builder, and that might be a way to not
> having to intercept its regular run. We don't do that right now, we just
> kick off dpkg-buildpackage on the unpacked sources.
>
> We also do that so far for packages that don't come as source packages
> in the first place but as debianized repos or as ad-hoc debianized
> sources (deb_debianize or local debian/ folder coming from an Isar
> layer). Those would require an upfront source package creation as well,
> something that dpkg-buildpackage currently does while building. Possibly
> cleaner.
>
>>> One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g.
>>> to inject patches before they are accepted upstream or to address issues
>>> that are not compatible with upstream packaging. Recent example:
>>> https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955
>>> (some patches not even accepted upstream yet but needed).
>> I do see why you need to patch packages. I do not see why that needs to
>> happen inside the build. Why not do it before building?
> As long as it happens in the same Isar build run, that is fine, I agree.
>
>>> This procedure should be automated and expressed with very few
>>> statements. The current approaches we have are far from optimal, but the
>>> vision remains that you just tell Isar the package, the patch, and then
>>> it does whatever is best from Debian perspective. That may include the
>>> mentioned source package patching upfront. That will have to go into our
>>> recipes and classes.
>> That sounds a lot like your requirement is a consequence of your current
>> implementation and not an actual requirement. I fully agree that such
>> patching needs to be simple to perform and (unfortunately) is a frequent
>> thing to do. Just the how we do it is sub-optimal both in Isar and
>> rebootstrap.
> We are not bound to the current implementation, just the "user
> interface" should not be needlessly changed. Means that the user
> provides source URIs, artifacts etc. via bitbake recipes and gets a
> patched and (re-)built package when building the related target.
>
>>> Sure. We are not building hundreds of packages, but it also not only
>>> about building one or two. The other reason is the mentioned identical
>>> versions in all buildchroots. Reproducing them must provide identical
>>> content (where common) for a single image build run.
>> I think I already debunked the performance aspect. For reproducibility,
>> you do want a clean build to reproduce the artifacts of a previous one.
>> Reproducibility needs to work regardless of whether you cache the base
>> image. Indeed, the reproducible folks are working on reproducible
>> installations somewhat and the dpkg-root-demo explicitly verifies
>> reproducibility (given the same inputs, but you do cache them anyway for
>> offline builds).
>>
>> So again, the requirement of caching an image seems to be a consequence
>> of your current implementation.
> In parts but not generally, as you pointed out yourself ("given the same
> inputs").
>
>>> Currently, we have no secure container env. The container simply serves
>>> as deployment tool of Isar's dependencies and builder-side
>>> configurations so that you can run it on any distro, not only Debian.
>>> And you can run it in CI identically to your host.
>> If your docker container is not setup securely, you can make it support
>> user namespaces and be done. :)
> You seem to confuse containers and docker. Our vision is unprivileged
> podman (or also docker-whatever, if that is still around by then) being
> able to build complete images. Still a long way to go, though.
>
>>> Isar users use Isar, they usually do not modify the core. And the goal
>>> of Isar is to avoid that they have to but rather provide easy means to
>>> configure things that must be configurable. If the builder would fall
>>> under this, e.g. no one build would build all types of packages (hope
>>> that is not the case, so far it wasn't), it would become an Isar
>>> requirement to switching them without much effort.
>> How would I disagree with that? Of course providing users with the
>> flexibility to switch their builder turns out to become a requirement
>> once users rely on it. However, the "without much effort" part is the
>> thing that mdbp is supposed to provide. In theory, your builder becomes
>> a shell command with a sane default and your user can change that one
>> command to switch the builder, because the API differences are hidden
>> behind a common mdbp API.
>>
>>> That is a valid point we need to keep an eye on. If mdbp solved that
>>> already, it would be one of the added values I mentioned that could make
>>> it relevant for Isar, even if not requiring to switch builders.
>> Great. Given that builder-flexibility seems to be a non-requirement for
>> you, I suggest that you use sbuild and look into the mdbp-sbuild
>> implementation to figure what kind of settings you need to fix to make
>> it reproducible. That's also a form of using it. ;)
>>
> Good! :)
>
> Thanks,
> Jan
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-12 12:29 ` Cedric Hombourger
@ 2021-07-12 14:35 ` Jan Kiszka
0 siblings, 0 replies; 13+ messages in thread
From: Jan Kiszka @ 2021-07-12 14:35 UTC (permalink / raw)
To: Cedric Hombourger, Helmut Grohne
Cc: isar-users, Baurzhan Ismagulov, MacDonald, Joe
On 12.07.21 14:29, Cedric Hombourger wrote:
>
> On 7/12/2021 1:47 PM, Jan Kiszka wrote:
>> On 12.07.21 12:54, Helmut Grohne wrote:
>>> Hi Jan,
>>>
>>> On Mon, Jul 12, 2021 at 10:25:41AM +0200, Jan Kiszka wrote:
>>>> On 08.07.21 21:34, Helmut Grohne wrote:
>>>>> The reason to want DPKG_ROOT is to prepare the root filesystem for
>>>>> your
>>>>> embedded board. If that's not what you want to do, DPKG_ROOT is not
>>>>> your
>>>>> tool.
>>>> We want it for both the board and the build env. We have the same
>>>> issues
>>>> to solve there, means installing packages in unprivileged manner,
>>>> either
>>>> for target arch or the builder arch (cross and native builds need to be
>>>> supported). So I see DPKG_ROOT as a building block for both. In
>>>> addition, it would overcome qemu-user, which is speed gain.
>>> No. Let us for a moment assume that DPKG_ROOT would just work for all
>>> packages in Debian (and that is a far stretch really). Then you'd be
>>> able to install your Build-Depends into a directory without using
>>> chroot. But how do you build your package then? You need to chroot the
>>> dpkg-buildpackage call. So you are back to requiring chroot, at which
>>> point you can just install your whole build environment using chroot.
>>>
>>> DPKG_ROOT is the wrong tool for that job. The one thing that it fixes is
>>> requiring qemu for the embedded filesystem image. Cross builds already
>>> work entirely without qemu (including package installation).
>>>
>> We can't cross-build all packages, unfortunately. That's why we will
>> continue to require native installations for some packages'
>> buildchroots, just like we obviously do for the target rootfs. If you
>> can create and also augment (install deps) those without qemu, you gain
>> speed. Granted, those continue to require qemu for the actual build.
>> Which brings me back to still having to add namespace support to
>> binfmt_misc...
>
> there is that:
> https://patchwork.criu.org/series/3855/
>
>
> I haven't checked if the author got any luck in getting his patchset
> accepted
>
See https://lkml.org/lkml/2021/1/18/1267. Since then, it's on my long
backlog, but with rather low prio unfortunately.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC] using lightweight containers instead of chroot
2021-07-08 9:07 [RFC] using lightweight containers instead of chroot Cedric Hombourger
2021-07-08 11:38 ` Jan Kiszka
@ 2021-07-26 13:55 ` Anton Mikanovich
1 sibling, 0 replies; 13+ messages in thread
From: Anton Mikanovich @ 2021-07-26 13:55 UTC (permalink / raw)
To: Cedric Hombourger, isar-users; +Cc: MacDonald, Joe, Baurzhan Ismagulov
08.07.2021 12:07, Cedric Hombourger wrote:
> Future RFCs
>
> - per-recipe buildchroots (mimic sbuild)
>
As Jan already mentioned, working on POC of sbuild integration was
already started.
It will be based on sbuild and schroot usage, so sudo for schroot
configuration still will be required.
We are still somewhere in the middle, but I've just send `[RFC 00/15]
Sbuild/Schroot migration` patchset to show how it can look like.
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-07-26 13:55 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-08 9:07 [RFC] using lightweight containers instead of chroot Cedric Hombourger
2021-07-08 11:38 ` Jan Kiszka
2021-07-08 13:52 ` Helmut Grohne
2021-07-08 17:16 ` Jan Kiszka
2021-07-08 19:34 ` Helmut Grohne
2021-07-12 8:25 ` Jan Kiszka
2021-07-12 10:54 ` Helmut Grohne
2021-07-12 11:47 ` Jan Kiszka
2021-07-12 12:29 ` Cedric Hombourger
2021-07-12 14:35 ` Jan Kiszka
2021-07-09 15:46 ` Cedric Hombourger
2021-07-09 16:12 ` Cedric Hombourger
2021-07-26 13:55 ` Anton Mikanovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox