Re: [PATCH] initramfs: Add missing umounts after generation

["'Jan Kiszka' via isar-users" <isar-users@googlegroups.com>]

On 03.10.24 18:15, Baurzhan Ismagulov wrote:
> Hello Jan,
> 
> On 2024-10-02 12:28, 'Jan Kiszka' via isar-users wrote:
>> Lazy umount was apparently dropped without review/testing of the safety
>> mechanisms and the code that is involved in mounting. We all assumed to
>> world would be fine while things de facto work not that well. On top
>> comes that most users are in containers and only notice those issues on
>> rebuilds or due to further special factors.
> 
> I'll share a bit of background on this: This is not about assumptions, we knew
> there were issues (and worked on them -- e.g., [1]). The occurrence rate was
> several times per year. After migrating to sbuild, mounts moved into
> per-package schroots, and the errors became even more rare. Debugging cycles of
> weeks to months are not practical because you can dump the mounts or similar
> stuff, but the necessary number of iterations is not reachable, not in
> reasonable time.
> 
> Before sending the lazy umount removal patch, we've checked the archives why
> "umount -l", "umount || true" were introduced but couldn't reproduce any of
> those (I personally found the problem descriptions and root cause analysis
> quite spare and often not actionable -- I think we can improve here). Even now,
> the issue seems to occur more frequently downstream than in meta-isar CI. Given
> that the downstreams have control of meta-isar revision and can also apply
> patches, the question for me was whether to keep the problems under the rug or
> work with downstreams to analyze them. If you have specific suggestions what we
> could do better the next time, please share.
> 
> For this specific occurrence, if we don't have the races with package building
> left after the sbuild introduction (this is why mounts and umounts were
> asymmetric with buildchroot), the latest patches might be enough without
> central mount / umount tracking or moving the mounts to individual schroots
> altogether. We'll need to understand the failure mode because otherwise proper
> testing is challenging.
> 
> 1. https://lists.isar-build.org/isar-users/20210421145855.66257-1-amikan@ilbers.de/#b
> 

There are multiple things:
 - mounts left behind - several, easily to review manually once it was
   clear that those still exist (I wasn't expecting that)
 - left-behind mounts detection - apparently broken, not properly tested
 - races and other conditions actually stumbling over those mounts -
   rare, specifically with the few cases of isar itself; and that's why
   the lazy umount removal appeared to be safe

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b2b32e26-c713-408a-8422-1354a48dc517%40siemens.com.