From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7010687509260664832 X-Received: by 2002:a17:90a:df8e:: with SMTP id p14mr4047576pjv.140.1632386343064; Thu, 23 Sep 2021 01:39:03 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a65:6487:: with SMTP id e7ls2080629pgv.7.gmail; Thu, 23 Sep 2021 01:39:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxvR/5PbnXzcRgKGuzMeRXZ8OQUOFcI4J2fWHW6PklQ23yacTxsKSAbIp5x+PHcwkYJXS8r X-Received: by 2002:aa7:8ecf:0:b0:43d:5046:2085 with SMTP id b15-20020aa78ecf000000b0043d50462085mr3079263pfr.48.1632386342460; Thu, 23 Sep 2021 01:39:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1632386342; cv=pass; d=google.com; s=arc-20160816; b=bxo8R+Ay/b9H6EA4407MthE9tZTXJ56PMmnQnInaGiYbj31D7+7Ky8E+qlufsHwNg+ T1UTXN+G8PmqjpuvkwMTwbT2bpRHWIC4CaqJKLvgp2RazrbIySX2Zes7gX9H0+I0bfpp LNrbChXdSskoxZ1+MHegETSdDgZYvn+nx33v/wwkWuD6qaTTXNN+BdB8EB6Oo1eAJd1a dsQhj1ACQ1/8fs8OyLkZNU26zogcF1i3V2o6DNpZ9ONZDCrtgbiEPTNCwerArTF/O54f iaVT2NwqgK24oaiiAQ129+kQCw3WsZiE8iS1CWIZBD3ipiv/aueGfDfXLiNhvQ+6Fvju A+vA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:document_confidentiality :msip_labels:content-language:accept-language:in-reply-to:references :message-id:date:content-class:thread-index:thread-topic:subject:cc :to:from:dkim-signature; bh=rTJoBaiGugo03mtnoE/kHvjU6rfcBZ9/29fyfQ2w/1g=; b=ORG7w2H8JDSqC4d0Whp1PQX+MbUIhZSyPE60HmRIQnpsLmZTF/DSWNG8f7wNHSCdJi 9fWt1wR1dSyTddpn7XKLrxpKe1J7tfZOjaqGXYioAWgA972we3JGExvKEHhGYay+sISs C2dIyf4M57xuqAVx23Aj4+DBu9NeTiZ6NH4Er8DLhh/JJmxoNZyfEbnD1zDmeLgVu/bM cdISikdPOiaHMqxxGeXN3FDxqDExmLyiuABOkAieGTEDOdeKs4ALT3p43YQ6ACqMMnj3 vWgKNjfeWzvM3L5JOD9SDENcfkw+WfkET1s8dABRKb1VRvqYeGp+JqJoUQdp5YKfG+hC Z/3w== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.onmicrosoft.com header.s=selector1-siemens-onmicrosoft-com header.b=FdZttljT; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of felix.moessbauer@siemens.com designates 40.107.0.67 as permitted sender) smtp.mailfrom=felix.moessbauer@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00067.outbound.protection.outlook.com. [40.107.0.67]) by gmr-mx.google.com with ESMTPS id v7si539907pjk.2.2021.09.23.01.39.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Sep 2021 01:39:02 -0700 (PDT) Received-SPF: pass (google.com: domain of felix.moessbauer@siemens.com designates 40.107.0.67 as permitted sender) client-ip=40.107.0.67; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.onmicrosoft.com header.s=selector1-siemens-onmicrosoft-com header.b=FdZttljT; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of felix.moessbauer@siemens.com designates 40.107.0.67 as permitted sender) smtp.mailfrom=felix.moessbauer@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MEx+jHYr5G+OEjGyQceb9W0OglhBNrJ6CVWL52cdVLMqlaBOduGkkGhDRqbaSCCaJytzvi0HhOgUMjuvJ8QgYQUb8K7JAZ6tDQNYIb8bRjvKxEmTTgjB4KaZ8ceMwZ3AQY8cCMD0r/m7HIujCAcoIMVtZtjv7GJXxb1JWM0D9FDQ5xCIDGyVxKKkE01c9/IJH1+49n6Ht3U7LqiyAbEdOaJuY+QpmMkLF1uDEcRxTWcBHrMRqtBnm32HeLWSQOfjjmY4whEIfLt0TbCBDccdUwS87eyDc+5OkHdPW7LUU7n+lpHA8o3frng6DIDsiM1KXn4Ck6MkRICj2T088HSQyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=rTJoBaiGugo03mtnoE/kHvjU6rfcBZ9/29fyfQ2w/1g=; b=li7dcNwltHB8DFem6eZclWk6K/JR74EFkJ04I44Ykh71ZNnp2B3QcybGxwbJR2ybYb+5vrhRjiX/qOwzIPixs9C31lm3HD2lMoL2+z+K+Nz0zBR0RBFKZ3WPhukManZxSZZiF+APWNwzWKyTiwporJATbeq6l9m/jC60MdPjzuNFklstWVyy0UZtcUxM4QE0KyWa5SEF3M7BUc19CcE3I3jHBF0sX2V1YzRctErW0gZN3IOw9gssx6MpltjGAcrTv62TchI3Vw6DyP+Spq8bNFnZmZoMLQMo9kSoYtkIDsrsq9q7ldNjVBaNtreSxRaq4bMqJTctDKP3D8dzzsL3Ew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.onmicrosoft.com; s=selector1-siemens-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rTJoBaiGugo03mtnoE/kHvjU6rfcBZ9/29fyfQ2w/1g=; b=FdZttljT31cYlWHYXR09PCSA3Jb/0f2qb7das1vJVK8l6EMKSSs9mlnSOgCg0i3wkPPhAhJfzAr1PyUD0AhoHOm+BAjdqaUCGx6Qo5Lj4qLn0mhFGfC8IyjFEm9c2vgFfiWnmKlvm0FeG49rXaLYIVj1iaQDtX2LBk3zHax5tr4= Received: from AM9PR10MB4869.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:418::19) by AM0PR10MB3282.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:187::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.15; Thu, 23 Sep 2021 08:39:00 +0000 Received: from AM9PR10MB4869.EURPRD10.PROD.OUTLOOK.COM ([fe80::e1e1:4c31:bcee:6345]) by AM9PR10MB4869.EURPRD10.PROD.OUTLOOK.COM ([fe80::e1e1:4c31:bcee:6345%8]) with mapi id 15.20.4544.015; Thu, 23 Sep 2021 08:39:00 +0000 From: "Moessbauer, Felix" To: "isar-users@googlegroups.com" CC: "jan.kiszka@siemens.com" , "henning.schild@siemens.com" Subject: RE: [PATCH 3/4] fix race by serialize rootfs_finalize and do_wic_image Thread-Topic: [PATCH 3/4] fix race by serialize rootfs_finalize and do_wic_image Thread-Index: AQHXr5QsRjLN1tcj10+wYPs8ircTgKuwKhiAgAEgtXA= Content-Class: Date: Thu, 23 Sep 2021 08:39:00 +0000 Message-ID: References: <20210922092754.3649703-1-felix.moessbauer@siemens.com> <20210922092754.3649703-4-felix.moessbauer@siemens.com> <20210922171507.40a81eb3@md1za8fc.ad001.siemens.net> In-Reply-To: <20210922171507.40a81eb3@md1za8fc.ad001.siemens.net> Accept-Language: en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_Enabled=true; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_SetDate=2021-09-23T08:38:58Z; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_Method=Standard; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_Name=restricted-default; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_SiteId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_ActionId=9fb12280-8205-438b-9dc4-d80f4068f3fc; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_ContentBits=0 document_confidentiality: Restricted authentication-results: googlegroups.com; dkim=none (message not signed) header.d=none;googlegroups.com; dmarc=none action=none header.from=siemens.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: aafd06d9-d23b-4e02-0417-08d97e6d9758 x-ms-traffictypediagnostic: AM0PR10MB3282: x-ld-processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: w9++7gRUy7ml3pFzTLmvcxn4QEEwDmcnpedM85BWPUOkYlf0fd7zZcNe3OszVTwBlBGBeG9+ysY4//SLz8mZhqmhNI4x5z83CI4bFMdmXtY1tRYXCZdtRbMFFvW+Kqu9olIvCdBxJI8v0HDOzmxKtXqUgeNDI2L6cmKM/gQWKe++HZ9egD/f02Ycnb+Jg0qfsxQbdbvz6K2OWRDaHR/G7tF8GUiajcgZ+jHMWRztmyKXQ5PxB7dG06JwdFXTamvRoyd4+UXGHH65L7L1/QRVlhnR3qIKLyGvDQKhIXuEGqVP2V98nG27PVts5FFtygRsr0DO2vAmXysvHUxn+PH8ArEiNtcXF4ZPiIXEEb+nHbuiWLRQvdLF2mmdxkwtayW6Bqbou/zE7HEx9G56oR8mwfEuBCA5eVqYAEdFIg+O0aAENybA3N0iBkm6rwq9FaqVj8K8ZQsfUvmrzR2wRnYCQUVnnKGUnJ9/sW7tXcFzJ9jarPv4YFq6Dwgnk3CHOBFoGgFpWs36LOm3M9jMh7Sz1Ng6iFyScAo3MNRTIVp461mV7Av7f99jxprG0KIWTx4wwkMVpygxJfhUjRXIq74ma7svycQP58E131DxWnWutU9LIVIm7KKZhHCSctxiMbujUOGKA0rVad8zK7dl8NZztXQb5A6/vEuW9gwq6wLBhthxo6Tn+xMRyzoZwSWl4og1t2ZIx6WV0NCq+s2nIbl1pg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4869.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(4636009)(366004)(6506007)(71200400001)(186003)(38070700005)(26005)(4326008)(7696005)(8676002)(122000001)(2906002)(38100700002)(107886003)(83380400001)(53546011)(508600001)(33656002)(86362001)(5660300002)(66946007)(76116006)(316002)(55016002)(9686003)(8936002)(66476007)(66446008)(66556008)(6916009)(64756008)(52536014)(54906003);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+Eh7n5ZflxoF1qnRlIMFqy9Ue33uaEMlC8WzJ9Twu4eeEvqhKx9Homyzgju/?= =?us-ascii?Q?KnU7xFz1Fh2tvsFPSINbzBVAeB4d2GjJLvHnq9rCIhKmazH+tC/Rgw5NADGy?= =?us-ascii?Q?Z/FHbSTVVNor/GpGJySDlmqvumJ2NLAy1S5EeEixRWZ/sF/2FMwPRaiRLskQ?= =?us-ascii?Q?YWnWard0XrE6wtYvo4sTqQOu68R9pVMDos3PlEpV+/Rw5HXm68um8QXcp6qr?= =?us-ascii?Q?GhEoP38oZf0pWO4IWwUHA0jkYoKGFPxoeTLs5KPaCL/icLXCOTMfoTntlp1A?= =?us-ascii?Q?0PDqENXx5s2kREHPFONDvOnSEKxbunN9IeZetlUJCt3sMNpnsc4l8yqMxuar?= =?us-ascii?Q?pybiCkMDKJ0lvI1nckihbThSOjq6Zv5DWVhSbCBiyfENv7bFDBJCKHltiooV?= =?us-ascii?Q?m4SBVPTEPBqjNCnqNzT/68ih1h+vqD8bkiAxBwXe3/AKUBA3EKF08eFBbuD/?= =?us-ascii?Q?Bm+cfAzVlC7d8Kvg1WG3NZ2VRczpoYNlSzvOC8Ecov16LP/AdqD2Bbhg6GJC?= =?us-ascii?Q?YSMIYT5N0GMjKYIo7W/bZ6OifgJDgUxIWzVWK+LQSSf04pzpYFvxIV1MFbje?= =?us-ascii?Q?cnrUMI+jW038LNWiD5/kvp1zBuWRtovY5vCni1q22znwND0eYkrxUaXUK+p5?= =?us-ascii?Q?gNDMYzp+7fYQS2hKJZh77ez7chbgkHg069SeLKaOM/WL54WbFGJezsrzodfz?= =?us-ascii?Q?NfDvy5U4TbI658K7mbg6m/rOM4a7kv64joD5lMDIkYwGBnnGGJL1RDBlX9a0?= =?us-ascii?Q?4JwCkNB5VGCpsBWFPOIuJkHa80CipnGVuXvZT2IlNx3LbIGWGJSFs3RGm/8/?= =?us-ascii?Q?qjC5key3AYl+/uhs2YvwVtuwICPppzbTrpOErMhl0q9zJlff6RKPA068iI9S?= =?us-ascii?Q?UUb/CLB4LUZUlN0j8KuOlNBjm/arYw/o7ZQxS9RShdiyfh6L0Cgff3n6EBcp?= =?us-ascii?Q?OkVG+g1UGx+pCb+BO3qgfLbnop80vXtxCgvUjX+04z46CJ9cUYhMqDU+tnY/?= =?us-ascii?Q?KIExenLBChCOrsSO3U/WuFpSXv0PHNV+7ZXBkWIMGFV8JrwRMilQ6Y1dYOMR?= =?us-ascii?Q?xScmJrpu9LT7jahggWQlOxsbYmsewybgNs8cL8y3NM5Sn39jshPjF7yf99Rp?= =?us-ascii?Q?++pKRODG/xUnCp5LIAQ4qsCvoyFLcGWB+UMIXamcMyOu6Jfru2kUIbx/2Z3D?= =?us-ascii?Q?G++CQ9eroDFe/fS/T8e4RwuMUyzlyQmgHtC3PmB+2Re8F/C4NYBmLs/lFqwc?= =?us-ascii?Q?dB4fVOt7jnGqnNr1LTGMCzAjEDddlVTzokwxQWurpJ1E9x1qn9wSV5GrVlil?= =?us-ascii?Q?JAZRGg+3bxDPpRa9UD95h/jC?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4869.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: aafd06d9-d23b-4e02-0417-08d97e6d9758 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2021 08:39:00.4467 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Ky+uOCvrqE0fTu5vOx4LrVmjtr458L14npkDFsFdLeDl4bQkuN+YeJa38niMAWANxAQEAiCnfkCr+BQVXAlycL4hAQDssl2KmYLc5MKlkzM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB3282 X-TUID: AJDoOj8WnEIy Hi, I had a second look at the task graph and this patch is wrong / redundant. The do_image_tools task already depends on do_rootfs which itself depends o= n do_rootfs_finalize. By that, also the splitting of the finalize / unmount in patch 4 does not w= ork out. This likely explains why we get a deadlock in do_container_image when build= ing a container image which is then included into a regular wic image with = this patch series. What really helped is to have a look at the task dependency graph which can= be exported as dot file: bitbake -g recipe-name -u ncurses xdot build/task-depends.dot I'll send out a v2 soon, containing just the first two patches. Best regards, Felix > -----Original Message----- > From: Henning Schild > Sent: Wednesday, September 22, 2021 5:15 PM > To: Moessbauer, Felix (T RDA IOT SES-DE) > > Cc: isar-users@googlegroups.com; Kiszka, Jan (T RDA IOT) > > Subject: Re: [PATCH 3/4] fix race by serialize rootfs_finalize and > do_wic_image >=20 > Am Wed, 22 Sep 2021 11:27:53 +0200 > schrieb Felix Moessbauer : >=20 > > The do_wic_image task requires a mounted buildchroot. > > Prior to this patch, the wic_do_mounts task and do_rootfs_finalize > > were not ordered. By that, the do_rootfs_finalize task could run in > > parallel and unmount the buildchroot. >=20 > That is not just a mounting issue. The finalize should be done before any > imager goes to create an image. Because finalize contains quite a bit of = "rm - > f". >=20 > If the actual imager task and finalize have not been serialized, we need = to > look into that for all our imagers, not just wic. >=20 > The pattern we use is that imagers usually do >=20 > addtask _image before do_image after do_image_tools >=20 > and it is also in ubifs cpiogz ext4 fit container ... >=20 > We need to touch all of them. Or play a global trick like >=20 > addtask image_tools after do_rootfs_finalize >=20 > to catch them all and possibly also catch the ones in layers. >=20 > Henning >=20 > > This is now fixed by adding the do_rootfs_finalize as a task > > dependency. > > > > Signed-off-by: Felix Moessbauer > > --- > > meta/classes/wic-img.bbclass | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/meta/classes/wic-img.bbclass > > b/meta/classes/wic-img.bbclass index e495c12..573537c 100644 > > --- a/meta/classes/wic-img.bbclass > > +++ b/meta/classes/wic-img.bbclass > > @@ -147,7 +147,7 @@ python do_wic_image() { > > bb.build.exec_func("wic_undo_mounts", d) > > bb.utils.unlockfile(lock) > > } > > -addtask wic_image before do_image after do_image_tools > > +addtask wic_image before do_image after do_image_tools > > do_rootfs_finalize > > wic_do_mounts() { > > buildchroot_do_mounts