From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7156869355346591744 X-Received: by 2002:a5d:6701:0:b0:22e:6545:995d with SMTP id o1-20020a5d6701000000b0022e6545995dmr12307669wru.301.1666357570259; Fri, 21 Oct 2022 06:06:10 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:5441:0:b0:3c4:4723:605c with SMTP id p1-20020a1c5441000000b003c44723605cls941408wmi.1.-pod-preprod-gmail; Fri, 21 Oct 2022 06:06:08 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4gZ9UPvrf6aFNmPpQvXnXDBOGU1u7qGOS7pU4ONiMfIeIf36ed61/dpXr2gLTr0hszTZbG X-Received: by 2002:a05:600c:3848:b0:3c7:1832:2b6b with SMTP id s8-20020a05600c384800b003c718322b6bmr3084921wmr.28.1666357560695; Fri, 21 Oct 2022 06:06:00 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1666357560; cv=pass; d=google.com; s=arc-20160816; b=CRcU/+o+JTyXKLbXovvOeJqME5L/fUsSTYYCF0hZ/uTcZyGH/GNI9oSnCRGeGOtKD0 CIZFK1nIIBirtUI65L+WlYNgSqfW+ObRXyMj3lAWfeaL7+2BhpoBXggilzKBmAUKqR0m ATbjhhiM/N+5POKxFJLcekRJKeNeCz/HFjwMX9tB3DCVE/oJ2Asc+dBTHftHxZjZXPCr rmZq88EeN9nNUDI+t5LYcQR3y6kTHOC6UF6UWIaKkPQaabNygjuBHj/gA138NZSXLeZP f7EUuUqI1TMJGCg4MLQhwEEwjWPRCb8MYgmA7Nz/ru59b6fPwQPg4Kf5kOaHHscBYb5O ufgg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:in-reply-to:from:references :to:content-language:subject:user-agent:date:message-id :dkim-signature; bh=MndpVmzKIa2ydnM7Mc0lHFawW1f2iIV7/tqtUnKn8tI=; b=qR+5GK3AwPaxOkafdvSsn9anYKomS+phxB1JMyTKpLFjUnpcvD/FwPoi1fDoR8bs5K psUtMerKirlBX1VDFVq1yTnWsa8pPHoVoAeiXzn/0hwncgvEeAGU2fLUi42QrlphIeYx csdhhn4TQeM5AVtDCOujobazePWE+4jcIOAAMmk5q432hMTyCU+icbPz+T5ig30q4hKa H1d1mvhGk4DyroGTtw2g9SuQGr1HjVueDdZZiCmkYofOOguqLpRlOLRb5vDQ1zz4E9pV 9GzzQ6LvEl8Ws/j7yzEGMLnT/mO9bb7BJLLBKUE4xCW+h+gsy/bgUI0htHLJ+s00WNER Y+aw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Znc6is9N; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of jan.kiszka@siemens.com designates 40.107.14.47 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140047.outbound.protection.outlook.com. [40.107.14.47]) by gmr-mx.google.com with ESMTPS id bp2-20020a5d5a82000000b0022e54ade3fcsi905717wrb.1.2022.10.21.06.06.00 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Oct 2022 06:06:00 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 40.107.14.47 as permitted sender) client-ip=40.107.14.47; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Znc6is9N; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of jan.kiszka@siemens.com designates 40.107.14.47 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iVW8BgKv6jQzfFNKW+OySQGRRokczqn+eH93ATzMOyCI1qsHgZcUqrD4GDylR8ZBmcEWONuS5OEWFEg7epkduGAkd0Tz5/7apkF6jc9PKPV2Q4eWjHKs9xOTbKreaKzlgCiUxtixGRqWUcbzReW0MyArJViP2X0FZJ9FSEyRONFeJ/gViDcFPfIPN1mLnlJSHSdAsJUgdiGj2rjhEdWwAZ5eaevxM7TaPMMYiIRqPtUZqbRdVDdIzVXUpeE66gcsSOpLrnNh4m+FC9w0i0v9KEOwOiZqXvj3k5M2noE7eEah27yPh9EOoL/Vb8FvU30Ud4Aw1ny+Y01QEuEB+wXihQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MndpVmzKIa2ydnM7Mc0lHFawW1f2iIV7/tqtUnKn8tI=; b=ZAbKiOp7jIYV2NRhQL8T60c+8fh3YQFSZQzZjamwtePMKb4/uOVBRQ623JWm0C6vw78zmWJ07vcRTwR7uTRAWvNUnRJTvLPwEj6TMAWkTVNQCXeuTdDc5itBGpYR3sxUommxNyT5LmjzMEtKvDJRl9FxDvPngpMV0pkDknb5VPGoFPZDY53U56Ke+8xQbx3RjK4EbtEzXw2OkbZS/bWIGCJA2oLGkeItZny4R2JsSogZt3fR9g50VELWGLthvWl/BcxND3/RNmygRnkXgWSMX239JLbVPL4W3gWhLBGciPpPV/2tKRYLDHnveS5ZLDKpPW6yZNEVY603+5iO4ndykg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MndpVmzKIa2ydnM7Mc0lHFawW1f2iIV7/tqtUnKn8tI=; b=Znc6is9NlXtELGRiIYavQxRnnqlpLjy7VvLIdcAQmdv+nelD7cvOQGqvT73FtSV5h82nqXv4yxcYTkV4o7nfOCoPzxT6GOc8xgcCRCoiNdPOR6HNQw5EA/wUxgEKLszv+B3LH++r7bB46SvZoasCsEvcY32/HYxsIoLq6kmSqjJGvHwwhJho8KvS4BU/KYRl2EoKzoJf7GHuqoqokT4LiLO7YUoGK3ueJdWLchQsKNdXAfKZIWRfXXZ7ez0wQVZ0VA8LPBbw8kHMIGmLy08cSD5Ru7v+KCVCsZ/6KQHUX3M2nS6C8co94v3kGVvKRCJwRbiFcAiUJIMTFW9q3bW9kQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:588::19) by AS8PR10MB6868.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:5b5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.26; Fri, 21 Oct 2022 13:05:59 +0000 Received: from AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM ([fe80::10b2:2818:6ac:68c5]) by AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM ([fe80::10b2:2818:6ac:68c5%7]) with mapi id 15.20.5723.034; Fri, 21 Oct 2022 13:05:59 +0000 Message-ID: <24d5c73e-9a74-fcd6-f6a5-0ee86a95e095@siemens.com> Date: Fri, 21 Oct 2022 15:05:56 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.3.0 Subject: Re: expand-on-first-boot and surprising umounts Content-Language: en-US To: "Bezdeka, Florian" , "Lisicki, Raphael" , "isar-users@googlegroups.com" , Quirin Gylstorff References: <6f036245bbe00d69fc3492a1ffbf91a862c2951c.camel@siemens.com> <06b9570d-b711-f325-f915-9f3612b0d6dd@siemens.com> From: Jan Kiszka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AM6P194CA0079.EURP194.PROD.OUTLOOK.COM (2603:10a6:209:8f::20) To AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:588::19) Return-Path: jan.kiszka@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS4PR10MB6181:EE_|AS8PR10MB6868:EE_ X-MS-Office365-Filtering-Correlation-Id: 58194a8e-323a-4c08-22a0-08dab364ff75 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qFr6xVpAI7QoRsnqzp83pJ/aBGmkOVRCmohqlMA3Tj9u1lXM5+f2tB5EpFnwuTZ0ogtZdRvDW/obyg+hsjRL0TSUqKN02Cw/ybW4kWtcIXUrM9pYADDwZQd9OixoBkAcdt+tKa3C0070iR3HkUxtOYax93iQ5+SRWNN6f3BFGoYPD8zKQC+8jmehGU8d7UAizPnMCtV+DZmaPOW/rjFAs2mWpCemiQmns0YlkVp7xqDqWZDGuilgUJG0xUzCITKbQEqiTrY/CfCHe2nKOWri0/SuRNTfIncAt36mE56Zaw8E8oPrOQCFYuapVaSYIrrdJlcLNjYo9+uJQVvneJ7tI4cW+OebrSDhPOfdMi6tJCv9mjfDXtMh/FWXHgi/JLA5Cw1G/aKx16XJbqr7el5SZwt9sr/faK8mN1nG33+iqI2KF0Y4FdaLnQ4K0UBZne5xeKkypZa2Us6/wnKNhReSgETzEGj1pU3396XWWy3rcX4tQ0jTktG0D2Nqn3pe5ap/tOC1C9/AGRTd1D4JVWAvtQILkJsgmS/RXc+hnDafdLc3CZYvf5h0YL8DZoaqbDQVS3bs8oa4Kwbe54TyF2oiRZIT2B3TjFYByDT9t/bhjW7Wn1Mc1KAMFsO/RPzTdjPV5fT0J9IAgFq8T9tdMg744WYt13yU2woWwV3D2IugV3guEwNIlEs0HY5ax5VKp+kL76oPJk/omDT7PYFi+F4T7FToTmuTUj7WNAeSAQvhW+Vu+eU/0IVx2wVqKf81zB0cPiOf3cppXwrY108vf4kvKPGcYt6XvuH1ET8a0BK2lqc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230022)(4636009)(366004)(39860400002)(346002)(136003)(396003)(376002)(451199015)(41300700001)(53546011)(8676002)(478600001)(83380400001)(31696002)(86362001)(4001150100001)(2616005)(44832011)(6636002)(186003)(2906002)(8936002)(316002)(6486002)(26005)(6666004)(6512007)(5660300002)(66946007)(6506007)(66476007)(110136005)(66556008)(36756003)(31686004)(82960400001)(38100700002)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aEtFK2NIb2pMSTBnTXVMcVJ0djJtcm5ZcWQ2RkxRT0NyYTZoZWhlQUY0MG1N?= =?utf-8?B?Y0hFUmF2VlBydVhwdjliNWZwUy9qckx3Und0a0docGFxZFpJanR2ZDV2ZzI5?= =?utf-8?B?VUliQzJYVHlpa3Fpd3REZSthMlQ5dzVPQkpGajY2Q2FhUklKcVpxM0pOYWhl?= =?utf-8?B?Rm8xUndZbjRSOWc0UEViZC95N3JKSURuZDFqYW15ZmUwQmU0Z1BQekhLZWZz?= =?utf-8?B?eXBpdGZtVlNzNnFaa051MHJXZEVTR3dPcnlHNzlJNmdNM0REMFhROWxLTEtR?= =?utf-8?B?LzUyVXp3aSt2TiszbFV6czlVNXdyUUhMU05lMUZUZGEwcHM4RDM3bzYvY2RX?= =?utf-8?B?ajV5dzQ0YWJUcG1qcFUxdHpHVlk2eS9PaE0vVFBROERZWTQzNVY4UjhYcEh0?= =?utf-8?B?OHY3VWJKU3QrNGJjVEMraStVaHM3LzNONm0yR2hHU0twNnpja09IQmRtQ3Vl?= =?utf-8?B?d2RTMkpwYm9UKzBUNjZrajYzTUQ1VXhOZ0hpT2Y3THRmUS9LWWtZTDdBK21x?= =?utf-8?B?QVNDRU9GQmgralJIU3MraVpjcTdyRHlkdm14M1N5Tnp3Q3JzeTlmRVVpZCtm?= =?utf-8?B?Ni9yZ0tuVHdFZ3ZLUWo3bFI5MGFSYnljTldudTZiYk45S2VUMFpndW03aVhI?= =?utf-8?B?ZEZEM1oxVlYxWWhwWkxYMzVNNGlUMExPZlR1aytrR2FaYXE2TGpVd29rOHFM?= =?utf-8?B?SmlNdldpOUs2V2YwRHJwWE9BRTk3QnBmN1hFL1h0QmNLcHFiODQ1UWtPNy9W?= =?utf-8?B?Nk1ETFluanYvQUVrVExjRm42bTgwN2lMTWNaZFRYL0RsNEZ3Z0ZaKzU4RzBv?= =?utf-8?B?UDFGS2ZzUUxnOEI0RkdEV2k3R043TGVkL01wSEZ3TjBUcDFEaEpzU3BtdGtv?= =?utf-8?B?VmIzQkxINVE2TUxmOVh5TTllaWYrUzlGQWlnWFhMclFHaHhFWTRxVnVuVEdY?= =?utf-8?B?SDIzV216TVFzbDR0UTZhb21tTDNoVkdSTkpuYkE4OEtiVlFOeVczOWxBcFho?= =?utf-8?B?c1lFeU5BbEpwZi9LcEZ5ZEJJem45UUFNWmJXT3ErUnM5WjFTeTF6K2RLTHZX?= =?utf-8?B?YmZJS3JBVjZjR1dtdk1VcHc1aVAraXYvQ3VxVEQ5Y01Wa0tlZ1o1RG83elVw?= =?utf-8?B?MWw1WjJHZlVXZTg0TEozWWYzdG10c0c2cFdQUG9Cdlo2VGRSUGJkSVVYNitx?= =?utf-8?B?U2lXY2NsbTA3VWx4T0FjY2JYSzN4YjlubVgvWVBCWXB3ajh5UWx2R084Q2t5?= =?utf-8?B?VzdITWo4bTY1ZEtVU3RCdXoxTTVYSlJLRTB4V25aY2tySDB5b3JvR1IzSDU1?= =?utf-8?B?eGJrNENTbGxXQ2hoWDVKMCs3UzNZQ0JVeFlJbmVkVzlxeU5maHpwM0pieDRP?= =?utf-8?B?bnU0Vit3V25LYjVBcG9YUi9yOWFncm1LRU9VMlNLd3BkcnBFemxqK3BwZDA5?= =?utf-8?B?Zm1CV0tTWnEwZ1l2ai9MVEhiRjRpNkZ2Q2hTYmM2eGlvam1hL1dUNzN6Vytx?= =?utf-8?B?NWlxbXRsZ0VYcmVVNkNmcmw1UE5FTjVXZFV4K01OMnNQQTJhbGJwbkxPeksw?= =?utf-8?B?U21hVGtMejFLRjdXM1Q3UkZydmVpVCtkWUpXcU5NMlBiTEZqeGNqTmpweWsx?= =?utf-8?B?ZlU4MHZ4cFVlbmRoaWNwcUN3aWJvWGFEQVE0RTVhU3ZoU3Q4VXdWMUM1YXBv?= =?utf-8?B?c0k3ekZ5d0RKTTBhUjNKNElQaWpLaVJkMG9KUkh5bHJLejF4S3pSUHRaS21s?= =?utf-8?B?bzM0eDNQVmcvM3dYWHZVVmU3VGRMa3lSZTV1UEl2OE84Y2NqcmIrQVZCY243?= =?utf-8?B?TnYwL2laaVFTeEFsK1lraHJiOUtzU2xSbGxSd3pRQzBZblBUK0JJOXpaTDlJ?= =?utf-8?B?TUpFTlY2SkY5K08xQ2t0cDdlRlJtTlUxQkFOR0Q4czZ1b1hab29GQjUyekUv?= =?utf-8?B?QmpvSTlzU0t5S2tSV0tqWHh1azR4eFBMV2dYeVFRVitvdzg0emttM3lSZjh3?= =?utf-8?B?TzZvRUR4Wk5TRWY0dmh3QnAvcG5YUkowMCt6Y2NYMjUrZFVnaGpBSlRHcWxP?= =?utf-8?B?a0lIY2laR0t6dTZiaFVnZHo0M1REcjg5RDIzVzdtREdnVFdoTHVyNVZTUjgv?= =?utf-8?B?b0tmV2IzSUYzM2tvQUpoOUxwMS92MS9ac0NRaG4xOUtCLzlOZ3NmelRSN3Rt?= =?utf-8?B?SVE9PQ==?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58194a8e-323a-4c08-22a0-08dab364ff75 X-MS-Exchange-CrossTenant-AuthSource: AS4PR10MB6181.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2022 13:05:59.3438 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0HwRf0x/Qe8SyfDM8lRQz2jqYMGf8OpKzkXnOagYB3P/guH17x9RHfh5Wv4KQ64SIybl3vxYM3CeO7XWSC1Y1g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR10MB6868 X-TUID: nR6uJKnsT7jD On 21.10.22 11:41, Bezdeka, Florian wrote: > On Fri, 2022-10-21 at 11:36 +0200, Raphael Lisicki wrote: >> >> On 21.10.22 10:45, Bezdeka, Florian (T CED SES-DE) wrote: >>> On Fri, 2022-10-21 at 09:48 +0200, Raphael Lisicki wrote: >>>> Hi everybody, >>>> >>>> I am using a debian bullseye based system and use expand-on-first-boot >>>> to expand the last partition. It is not the root file system but an >>>> extra ext4 partition to be mounted under /data. The mounting happens >>>> after expand-on-first boot has succeeded. >>>> >>>> Sometimes, on some builds more often than others, /data gets (attempted >>>> to be) umounted immediately after being mounted and subsequent units >>>> will fail. >>> >>> Hm... Nothing happens on build time. Everything takes place on the >>> "first" boot (until expand-on-first-boot disables itself). "more often" >>> is also confusing me because expand-on-first-boot should run exactly >>> once. >> >> Yes, it runs on first boot, but I can use the very same original image >> multiple times (after restoring it). And I can also try the same with >> other builds (doing a first boot) and some builds seem to be more often >> affected than others. My guess from this is that the exact >> alignment/size/padding/moon phase of some things in the image also plays >> a role. >> >> >>> >>>> >>>> Removing expand-on-first-boot resolves the issue, so does adding >>>> "ExecStartPost=/usr/bin/udevadm settle" to expand-on-first-boot but I am >>>> not sure if this is only a sophisticated way of solving a race condition >>>> with "sleep". >>>> >>>> My gut feeling is that after expand-on-first boot finishes, the kernel >>>> still processes block device events, which systemd gets after /data has >>>> already been mounted and then cause it to be umounted, as systemd was >>>> already picky with umounting stuff in the past [1]. Unfortunately I have >>>> no idea how to test this hypothesis. >>> >>> Nothing should be mounted (except the rootfs) until expand-on-first >>> completed. See below. >>> >>>> >>>> Did anyone already experience something like this? The racy-ness of the >>>> issue makes creating a minimal reproduction hard. >>> >>> Nope, have never seen that, but that doesn't mean that there is no >>> race. >>> >>> We have >>> >>> After=systemd-remount-fs.service >>> Before=local-fs-pre.target shutdown.target >>> ConditionPathIsReadWrite=/etc >>> >>> inside the expand-on-first-boot.service file. >>> >>> So we should be done before systemd tries to mount your /data >>> (according to Before=) and start after systemd did remounting in case >>> mount options (ro, rw, ...) have to be adjusted. >>> >>> I guess you should check your startup order (systemd-analyze might >>> help) and report back if we missed something. >> >> systemd-analyze shows exactly what you described: after >> expand-on-first-boot has finished, /data gets mounted (and immedeately >> umounted again). > > OK, so you have to figure out why this happens. Hopefully there are > some logs available... > > What file system is on your /data partition? What is the initial size > and the full-blown size? > This pattern rings a bell, I've seen something like this on the IOT2050 as well. I'm still looking for the hack that worked around it, maybe Quirin recalls that more quickly. And I think that hack was obsoleted when we moved to r/o-rootfs and overlays... Jan -- Siemens AG, Technology Competence Center Embedded Linux