From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 14 Mar 2025 07:26:17 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-il1-f183.google.com (mail-il1-f183.google.com [209.85.166.183]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 52E6QFog029845 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 14 Mar 2025 07:26:16 +0100 Received: by mail-il1-f183.google.com with SMTP id e9e14a558f8ab-3d43541a706sf19144135ab.1 for ; Thu, 13 Mar 2025 23:26:16 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1741933570; cv=pass; d=google.com; s=arc-20240605; b=HnNbjY1LhV6ApWdiujSrdXrg+2/IcUp1GU21/eiPcUYoOqflvE0BCTjyVMeg3hcjpo duahoPpotQtlu7p3ntgLEE87OEOHqwYyNVTMj4yuIEFM3KO7m5f1lAtKtT9q2bapH41O cC8PDA+/Yag5Z4XLKZCCV8cPVpY6HzdPyIOZPFvwFZkGRZhlY9clHT/tEk4LejQClvRc qrHDA4FRvzXqhPOeBmm6GoYKM4GJ98laGg6Eg3tZFfsOHg6rEzJQi/7LRPBZE3WAdpxC 1mrZMrBFd3x39Zb+CGJR1W824mjLaqOqBRuneXKyJYlDOYy6J8D4F7iLtt7joUIS/aG4 hdlA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:user-agent :content-transfer-encoding:autocrypt:references:in-reply-to:date:cc :to:from:subject:message-id:sender:dkim-signature; bh=UscR+1GZqi0in2IsJ23+h483UyzqfHT98ATdN6xJAOc=; fh=TbPOO5PgulO4MOiDFAl1Pcr4IqSjwh6Vkiu96+YToyQ=; b=Cjlk1T7CJNLtrvIpHnZpZZRn1cSoPhjS9TBQH+A7WN+tqO0uePleYxpl+MnAHVd6bT gqascFkASo1fEskDcWYe+sjkFhuz7rjpv7TZOZoH6BoeLTp0l2X3EyWjRNyiEdgsmg04 Kuu4eOei23ghHr0vfB4xnAsuB76LIex2oToubxhl/ZjtbJMtL9A8PhJG4bM/X6OeGGao UtnAXhYy9o5XTW2dV7lG5z23jhdFB8XPzlqVMKjKqRZGvUgWmKtHn2asIdeWOEwllc7S E0L82q6lBHaRSDfg465J1gSOV3ckJ/TstWwErfb+mj4PxBHRA8gaTSH6SEOqTtQMOPih eyfA==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1741933570; x=1742538370; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:user-agent:content-transfer-encoding :autocrypt:references:in-reply-to:date:cc:to:from:subject:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=UscR+1GZqi0in2IsJ23+h483UyzqfHT98ATdN6xJAOc=; b=McEkqvWsUSDxsFtmcthPGJ72TBQhwHsl/EeBnVGEnI+19OkhrRUHXHCuLU39oKeFYD FeiIbowNZYy7Ee0deAqzcyFYbwZ+bDv3Sg8tmLxiBZQWcqBylXdkPL72I8qVEanw/E53 7eXin7euJ2eZfgXAJa7qOgSTk4WA0dPv8wkgqU04cI/uvC++Rlor7bWMc+XV57+O/37v 0qqzV3celb1YkR5mzq26PnOdSRLrJWzi9NDzvqc1W1MsJ0wWKFF3ZX14I+S3AsyElutV F98IAqGVGeU+KCEr2fWu0Yb9ziPgi91UILGFzk5ODlEd1vD/GOnYWHH3/F1oFlYBmKYh TtqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741933570; x=1742538370; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :user-agent:content-transfer-encoding:autocrypt:references :in-reply-to:date:cc:to:from:subject:message-id:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=UscR+1GZqi0in2IsJ23+h483UyzqfHT98ATdN6xJAOc=; b=aRYXEPrlXZBHwRnNfnUqgr8HReMx7PQbvx65F59rZWIAAlcsbQXtF9nm6sv4DV8lYT fUaBpdp6O4Sy4VOIJfg4aBm8LUM1jgHe8SH1BIEErOnzcKW54qKBQeMnpFHNOG21YX0D /N58toyxM1H4bYMC1fDDHydMFF38Xw/T8DsIURXTCr0fjkiet7mEuGPqK5Cu5jINPZUC bsDIVDMtKqua40gOqTSVI36Pd1cMaaSz9q/1DKbM1kFd6oGX6eT6DU8WUgVwZN/RWb8M 1myzGfRNmMWCgGXCaYaKn5yceSj1mhYSETswS1CG01sAYoB5LfnAe2FiKFM37Ygt3+It YUMw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUYkpQYrWGkqkT1NLwUf36d6+JHdArxlFwUfwnyjDnT7fID5fTGZmkkuoivHga5YvVSF8A7@ilbers.de X-Gm-Message-State: AOJu0YxjLl0sqYto1zuaztboHGycvckhPPhVA+A0hO20cff8SvGCTVNi 00d2s5pVyDNH9BFCcD9nNEDKhMt+rtDxycSQpv9Ui23GTLXhGy6T X-Google-Smtp-Source: AGHT+IEoUMuiG+NUDrMbWEAhGzuc6Sc5TJr448fVJMneLTD24ymQQpvLQiMg69NyodO8KmOuVDsuxw== X-Received: by 2002:a05:6e02:2486:b0:3d4:2362:98d8 with SMTP id e9e14a558f8ab-3d4839fea62mr13172865ab.2.1741933569683; Thu, 13 Mar 2025 23:26:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=Adn5yVFlS7amrjNw1CQ3IIn3c1BFKiwz52/zYDWUlMWkTWF1RQ== Received: by 2002:a05:6e02:3e07:b0:3cf:c8b9:882f with SMTP id e9e14a558f8ab-3d478837db8ls12960565ab.1.-pod-prod-09-us; Thu, 13 Mar 2025 23:26:08 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXDZZuCd7uh4glI2RVr7jsnrJuWkOsFl54SNrgM76BRb23t/A2WXYJFNkvxkSzopHuf3V64KCb/eui0@googlegroups.com X-Received: by 2002:a05:6e02:b27:b0:3d3:dd32:73d5 with SMTP id e9e14a558f8ab-3d4839fe860mr10518265ab.4.1741933568655; Thu, 13 Mar 2025 23:26:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1741933568; cv=none; d=google.com; s=arc-20240605; b=fsTqK60h3XZ8I+vAal8nAw8PJZpjjvXXJVj2H614/rl/2RL7F1PCvREOwyuBvgNzIy vEpdYleX/CKXgkUDo0AbD4TrcHWcShS3tIsGP/0EAhgG6cYnHYrjmZkhsJ4py1+NRpJY yf+0MfoxKi7Ge/bCmkv8sHI2uoCd3i831C2N4V3s4Es8eM9/cwWpBtrM/H5SEfPC479R 0hTfxZ3ATe50XOJnfo5qzXFHKLaoU2c3eK5+C9oIyB4FwsGmsYdyRBle7NwbZ0vRq5KX kNkmHnb7ntvH/XhtrSgcqUD9azkTzlgRxt2Z61DT/DumLtujTSLmvBE+xvsrfbd4kyaP y3iQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id; bh=vJ5oo1azpkBOmz13jIsReeNHSvd/PZdlMCEnLs37qQY=; fh=ckLJ/YdaA2J+MKRK+92UKmsLtjKnSsvrxhMy7erkAFU=; b=cZWtF3HRtfEE9vNcZq43PGl0S+8Saoo8MOIsNz2mZb2IdpB/p+4DyQihgg9ix1YjgX DAQsksWAZ40+C7/OfsQO0YZoVOFzh7aZ3Kg6FmDDtnympa92nLPafUS0PjYZCFl4Trcp aVqWkl/TCEwcnjP/aib1GSI618E/M5vbrT5bDX1Y7xOqYuI8S9U0BWD6GS6YNkStA/7/ kSUmRYL+4X27H5MH0MdAaSvXJRKCnyJYKda15AOWdXUW/7agyP9uA10bbBdTVMjW1+7I h1+B0SieU7E/aasLl0pQ1zoXrX5bnX0FN5kxWT1SvU6EFndWHkkVuN5hhlAjS1ukRub4 Tmig==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 8926c6da1cb9f-4f26372b693si134127173.2.2025.03.13.23.26.08 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Mar 2025 23:26:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from [IPv6:::1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 52E6PxIf029838 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 14 Mar 2025 07:26:05 +0100 Message-ID: Subject: Re: [PATCH 03/17] meta-isar: deploy-image: Change reboot logic From: Uladzimir Bely To: Jan Kiszka , isar-users Cc: Quirin Gylstorff , Shekar Kasturi Date: Fri, 14 Mar 2025 09:25:59 +0300 In-Reply-To: References: <6e97e882f080e09a9f7d777cc84e9cd089d92d11.1719927511.git.jan.kiszka@siemens.com> <3fd6bdc491628229686855d3278b8d5684643108.camel@ilbers.de> Autocrypt: addr=ubely@ilbers.de; prefer-encrypt=mutual; keydata=mQENBGO2eUkBCACtT+T3OrPVSExBmqfgXT3lp9XcdxRzjYp26wezkgYjjBXaf36bxtaAf S471VoQtpar0RVeFfW7WDDdfX9ZclSj36zBQe+RVSJzoNoNQfjOXWuSHb5Z+cpAFtqBY4muxK4+ia IlLJd6CN3ejOsLHATtCeHHq8wi0z2T+KdLQO+wQRgo2hjj0Lp9pGTrKJry50HP/o7Vbdu14dOx2xq r8+wPc6SQbBIrcqaa4MqCQC00vQG7eXvo+k2MOw59FDdpMH0KR9mHgp3u/s4I+4YRBArukt9G9xz/ rsEFmxAIBC6N/a6Hzwg4puc91n7ABDsPg8Vp+X3MDraujN0dvR6OKVNtABEBAAG0IFVsYWR6aW1pc iBCZWx5IDx1YmVseUBpbGJlcnMuZGU+iQFOBBMBCAA4FiEEJqPNVhVGyk12Eh+PAUQYBM/2FkoFAm O2eUkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQAUQYBM/2FkprlAgAmuna8Hm9EyoEtTl DBGDg6Zm4ZLp5ffvZBE946h92jepDrteoxsJ7pSzJVC2HmDLa4iZUao7lLLbDsUj5x45/iLJcqBZK k3YnAxP2r6a+kI+1VVQY1pxdG1nlJAbdNzoojm/qmezNPSrqni61KVMQKsXBCWhIjSXDSM9CsBj21 a+9qaVqfxovJGTn9lgrZO+xzKQNMKZeOouJlscVuFj21P0ww3/YENiU/nMeTSuYypO76mDtAd08Jo nc3yuHa9MJGei5ixN3wT+IrGR2aL2hdw2M6NgH7sYbL2Zi4ugD6RXHJai1Bh2yvFSVqSQ+M6QOInT 4ud7wslm1XRB065dXtA== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.3 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: AjL7Akg5xPN/ On Fri, 2025-03-14 at 06:54 +0100, Jan Kiszka wrote: > On 13.03.25 14:10, Uladzimir Bely wrote: > > On Tue, 2024-07-02 at 15:38 +0200, 'Jan Kiszka' via isar-users > > wrote: > > > From: Jan Kiszka > > >=20 > > > Pull the reboot out of the script. This allows for cleaner > > > integration > > > with different calling environment, may they be a systemd unit, > > > an > > > initramfs script or simply a shell for testing purposes. > > >=20 > > > And if the script exits with an error, wait a minute before > > > rebooting > > > the system, rather than just trying to re-execute it. This > > > permits to > > > inspect potential error as well. > > >=20 > > > Signed-off-by: Jan Kiszka > > > --- > > > =C2=A0.../recipes-installer/deploy-image/files/deploy-image-wic.sh=C2= =A0=C2=A0=C2=A0 > > > | 2 > > > +- > > > =C2=A0.../recipes-installer/deploy-image/files/install.override.conf= =C2=A0 > > > | 2 > > > +- > > > =C2=A02 files changed, 2 insertions(+), 2 deletions(-) > > >=20 > > > diff --git a/meta-isar/recipes-installer/deploy- > > > image/files/deploy- > > > image-wic.sh b/meta-isar/recipes-installer/deploy- > > > image/files/deploy- > > > image-wic.sh > > > index 8043aff1..12c1eea2 100644 > > > --- a/meta-isar/recipes-installer/deploy-image/files/deploy- > > > image- > > > wic.sh > > > +++ b/meta-isar/recipes-installer/deploy-image/files/deploy- > > > image- > > > wic.sh > > > @@ -105,4 +105,4 @@ fi > > > =C2=A0umount "$installdata" > > > =C2=A0sync > > > =C2=A0dialog --title "Reboot" --msgbox "Installation is successful. > > > System > > > will be rebooted. Please remove the USB stick." 7 60 > > > -reboot > > > +exit 0 > > > diff --git a/meta-isar/recipes-installer/deploy- > > > image/files/install.override.conf b/meta-isar/recipes- > > > installer/deploy-image/files/install.override.conf > > > index 73874caa..357d8662 100644 > > > --- a/meta-isar/recipes-installer/deploy- > > > image/files/install.override.conf > > > +++ b/meta-isar/recipes-installer/deploy- > > > image/files/install.override.conf > > > @@ -1,5 +1,5 @@ > > > =C2=A0[Service] > > > =C2=A0ExecStart=3D > > > -ExecStart=3D/usr/bin/deploy-image-wic.sh > > > +ExecStart=3D/bin/sh -c "deploy-image-wic.sh || (echo 'Rebooting in > > > 60 > > > s'; sleep 60); reboot" > >=20 > > Hello Jan. > >=20 > > This change seems to cause a sort of race issue. > >=20 > > Few days ago "[PATCH v3 0/2] Cover installer image with tests" was > > merged. Tests were OK (and "next" is still OK) on two jenkins and > > one > > gitlab instances. But on the third jenkins instance, they were > > failing. > >=20 > > Boot log: > >=20 > > ``` > > Got config: > > =C2=A0 installer_unattended=3Dtrue > > =C2=A0 installer_image_uri=3D/install/isar-image-ci-debian-bookworm- > > qemuamd64.wic.zst > > =C2=A0 installer_target_dev=3D/dev/sda > > =C2=A0 installer_target_overwrite=3DOVERWRITE > > bmaptool: ERROR: An error occurred, here is the traceback: > > Traceback (most recent call last): > > =C2=A0 File "/usr/lib/python3/dist-packages/bmaptools/CLI.py", line 116= , > > in > > open_block_device > > =C2=A0=C2=A0=C2=A0 descriptor =3D os.open(path, os.O_WRONLY | os.O_EXCL= ) > >=20 > > bmaptool: ERROR: cannot open block device '/dev/sda' in exclusive > > mode: > > [Errno 16] Device or resource busy: '/dev/sda' > >=20 > > Rebooting in 60 s > > ``` > >=20 > > But either machine is not booted, or there are some "ext4" errors > > happen. > >=20 > > I couldn't understand few things: > > =C2=A0- why this happens (unable to get exclusive access to block > > device)$ > > =C2=A0- why device is written even if there is such error in logs$ > > =C2=A0- why sometimes we don't see this error in logs$ > > =C2=A0- why this doesn't happen when "unattended mode" is off. > >=20 > > The reason is that we have these two "getty" instances overriden by > > custom one having the race. While "serial-getty@ttyS0.service.d" > > does > > the job, "getty@tty1.service.d" produces this error visible in logs > > and > > reboot machine in 60 seconds. > >=20 > > In some cases 60sec is enough for first service to finish writing > > to > > the disc, but on slower machines we result in incomplete copy and > > broken system. > >=20 > > Before this patch, when "reboot" calls were in the script, "failed" > > instance simply exited and didn't try to reboot the board. > >=20 > > As a quick fix, I would reverted this patch. >=20 > That won't be a fix, just papering over. >=20 > The race is apparently around the target device becoming ready for > bmaptool, or our script racing with some other service accessing the > device for a while. I suspect the service is lacking some dependency > here. So, let's identify the race partner first: How to reproduce > locally? >=20 > Jan >=20 No, my finding is that getty@tty1.service.d conflicts with serial-getty@ttyS0.service.d one. Every of them tries to run bmaptool when unattended mode is active. meta-isar/recipes-installer/deploy-image-service/deploy-image- service.bb: ``` do_install() { install -m 0600 ${WORKDIR}/install.override.conf ${D}/usr/lib/systemd/system/getty@tty1.service.d/override.conf install -m 0600 ${WORKDIR}/install.override.conf ${D}/usr/lib/systemd/system/serial-getty@ttyS0.service.d/override.conf } ``` It can be easy reproduced if we decrease timeout from 60s to e.g. 10s, to simulate slower system: ``` isar $ sed -i -e 's/60/10/g' meta-isar/recipes-installer/deploy-image- service/files/install.override.conf=20 isar $ ./kas/kas-container shell kas/isar.yaml=20 builder@b21a175d3b78:/build$ rm -rf conf/ builder@b21a175d3b78:/build$ /repo/scripts/ci_build.sh -T installer ``` Failed instance of overriden getty reboots the system while the target is being populated with the not failed instance. Reboot hangs at something like ``` [ 9.323801] JBD2: no valid journal superblock found [ 9.325117] EXT4-fs (sda2): error loading journal ``` > >=20 > > Also, I think this behaviour may impact on bmap-tools "psplash" > > patches > > currently we have on maillist. I would expect that user won't see > > the > > progress bar on screen because it's somewhere on serial console. > >=20 > >=20 > > > =C2=A0StandardInput=3Dtty > > > =C2=A0StandardOutput=3Dtty > > > --=20 > > > 2.43.0 > > >=20 > >=20 >=20 --=20 Best regards, Uladzimir. --=20 You received this message because you are subscribed to the Google Groups "= isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/= f56884186356c514f66cbea793b521640be7ee1a.camel%40ilbers.de.