From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7071655505424285696 X-Received: by 2002:a2e:2f01:0:b0:247:dc8a:990c with SMTP id v1-20020a2e2f01000000b00247dc8a990cmr6662263ljv.342.1646664823147; Mon, 07 Mar 2022 06:53:43 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:17a0:b0:246:87f3:d0e1 with SMTP id bn32-20020a05651c17a000b0024687f3d0e1ls2220197ljb.7.gmail; Mon, 07 Mar 2022 06:53:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJw6rp+GmXfPcXFn5NibyNvOsVOooBZSSEVQnc8W7qyuziXfLJnG9xbwEqTic56kHTn3XusL X-Received: by 2002:a2e:1544:0:b0:247:dce4:681 with SMTP id 4-20020a2e1544000000b00247dce40681mr6611513ljv.430.1646664822121; Mon, 07 Mar 2022 06:53:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1646664822; cv=pass; d=google.com; s=arc-20160816; b=DU5QqhB0ciOWejBN/F+scy1stXW8cVHXbKhPlcOpCt7JUBtQeBcEgu3GxAg7hNnx5j 6G03twGMBuEQ/EfQVRfLMI64Jbm3INyCbjwSzVV/9sJ39xclDLEqhokODPOcqn+4bOEB Ng3yXvi/V7dvgLT5ofDbkRkw+obtGAK3g8xbj4EO6mjIUSEKl3RbhUOHzhKdwghNPR9j q6HXtwBBoxZ8ZuClIc0DEYvI1MFykF35xHfKXjMB1oKrsp53LZA1rYXZcXKLLq1hy9nM 1L+i7mMlLwy29DH9ZTMNsKAManCL/VimclSqwHKhn3Jz8cTfNWJtxZ9y2XpDCyA7ld04 mEtQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=21OR+dU0dcix9/A1f+ic6z/8/XjY+gdgt3VCFxCh3CQ=; b=mgdhRjcSIQTuRJWDCZBXLFH5qvAwZrINy/2V8/QOEdQdlqtXpPC1JPqTKmLWv8PvqA 0v7V4776CHjan5ogzCrc+E6l8BJKl/YGoQVJF1tceW8LuiyQZgd5beGYQl/khStWrLro WkBB2vOClUN7bPz7E2PHXHRdDrUO8gVo7v5RKgRmf32Uw974qPXrgv89CKCsih29ixJJ PpxRk+ip0jUCkiIQAzC+OoCCFN9dBsNCqWdVW3yEBrC0RAgiOMBVgCHDzNuNG3guIfxj bUlSc1d44N/wFcoU0c/f+Fg3Z3jKxvgNxR1P70kO/EL0QQ1jFYeW4mzFrVqSWLYe6bUq vZAw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=qpd5MugA; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe08::60e as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-am5eur03on060e.outbound.protection.outlook.com. [2a01:111:f400:fe08::60e]) by gmr-mx.google.com with ESMTPS id y10-20020a19640a000000b004481dd0bdb7si359582lfb.10.2022.03.07.06.53.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Mar 2022 06:53:42 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe08::60e as permitted sender) client-ip=2a01:111:f400:fe08::60e; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=qpd5MugA; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe08::60e as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UjZYD25cwH/am7SI78gMOjTDxO2rauduNSFFdV9Z5XdS9jqURkyRrogV4om2VxHLabIrnEmnMWCHmrFZciglBSLrdiNoiZSeX+phkJeHzhdMISfmR87ohWioH04BhHgmtVZ5/vaiDTekYGT2riAzUXyUkFb2Zi8pEdvyfhVPD/m9XBPvdm57FXeELr4xP9eSluxTPUUUvD15WDaBypUWPy6dahhBRSJcQItw55i4TsGtwtVlRAwAjONF8kYHuFjV070AT8Po+viCJtkD4QB4ILA17kGuHCbW031SwWt//j6E5HbxYixBXr9Al4YKS5/jaqdTA7cJx0x4oGtQXH5SFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=21OR+dU0dcix9/A1f+ic6z/8/XjY+gdgt3VCFxCh3CQ=; b=Zkt2p7qX1D81WvN5ubzxvEj8IhuUMqjAbQyBW9BHiNkZ1Ino8rqpo067fTbWqikgKOdjIoDaCy1EmWKTiiNNwsKawnoon2G8i0GNXg2DGIqT8cWnZag3aRFO3VCXI6v4xw8upVKuHOPuJkwMoIEfybN36uArHzwQRkUQ/kFNJqoBja3niQlve5yPg/H8eTbLhvEiWI8HaySIeTC8vwvvkQ9kk5P9fcNEu5DT2RgZ7fi51i80iYGzAr/3sLOcM8EiCIX+cuZ1QYSEoy6tPvJYyq96Kn9aRV1HSBqwBvSc5QipJGlp9VoCOu1foSQdFzZcmaXAoaWUHYb6W7F1hp8Zcw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=21OR+dU0dcix9/A1f+ic6z/8/XjY+gdgt3VCFxCh3CQ=; b=qpd5MugAbahX22BKvDyZ9mzoCDSAeRupC3FELghnJQfNaIjsu1Icp5TvCnESwgS8oAyNxXYFtf+cvxehTQpQnb/3GQeMPkl+QwDsSA9yQQ1hCq+0ih0ZiQcnhDX2PaF9Io7XxonZaRDPDfudUJkDzIp53XbewWsb6wxvZQV3N33n347EgMhyKFwdJKbkV52UDfjE062PuIox9gTalwOpMaW7YboEj++s3RS6fVDS/2fhAk459G9D5Z9N4x2zKD3n6qsnCExt77fdXCsZ+NbT5KDWYjH5A5WkpP62lV8TL0O0VAH9u0zjpAGEAJzb2+yxKHgyykNRufTQDyUmi1GjAg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:13d::17) by AM5PR1001MB1171.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:203:9::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.13; Mon, 7 Mar 2022 14:53:40 +0000 Received: from AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM ([fe80::4855:c766:bd0e:ab46]) by AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM ([fe80::4855:c766:bd0e:ab46%7]) with mapi id 15.20.5038.026; Mon, 7 Mar 2022 14:53:40 +0000 Date: Mon, 7 Mar 2022 15:53:35 +0100 From: Henning Schild To: Jan Kiszka Cc: isar-users Subject: Re: [PATCH 3/3] vm-img: Handle sparse files efficiently Message-ID: <20220307155335.6f706439@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20220307100403.3ac04025@md1za8fc.ad001.siemens.net> <20220307150941.046b852f@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AS9PR05CA0035.eurprd05.prod.outlook.com (2603:10a6:20b:489::17) To AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:13d::17) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 172539ac-0122-437e-b191-08da004a445a X-MS-TrafficTypeDiagnostic: AM5PR1001MB1171:EE_ X-LD-Processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: r1A8H5l0OdFSjn9vDfDTRzLo8AKNPU+ntzVAZYB7f++vXwufw6cqqn9rB+RNmNXk/B/BIXB1/w7pxLgVqOeDiWbVQqSd6DshgLTb50kOf3TQzKOwGz+p9eE9q1LDvEiPMvRjq4l2lyWGxTmHMERwdUyIIJ3Xws4JBfJvlC4zrXkMwaNw0bXEZVn2Iz0wQ5J8aHYhdCfdSKk83V5v9Bhq6fyNPFdvlar0SwlZdlJ4Jq3eqoILtTJCmFI/EmRWzemEWQeFG23LEHgxeT/s1mkc8GUa2JjlKwLQ9TknscdOU22L9ppR+7JvvTvYh95cSQxRwfQzApjFVUqhNv/VNtGCDUwkv837UupwulfHLCpDqe58KDVRs+pGbJfm2woEvGfqW4Hf9h5+3Zb6LzHywNTmVf8I5Klwe4GoB2UMx6dRIgqdDJSESOwjAjbnizdkBB5wpzNWu62wdwC89/NHOsh7rEEfaftLmINtuoopNsDU0WLIHBqJ44ikHvX87EuhKkSTkrp/XqRLZ7YEgIH5mItD/mZHtbMd7onZDMn3bpWilXuZUml4X8tiqpF5YpbEaPgo/OMY2YIfkQZ7oJoiOy15sj6tYDuV1UCY5Z1pDRJyUO+G8ZoTio+WHOZzuzvW1y7kl8JIPDQcDv+4deTPVgCeqQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(8936002)(6512007)(9686003)(44832011)(316002)(66556008)(5660300002)(66946007)(26005)(66476007)(508600001)(8676002)(6862004)(4326008)(86362001)(186003)(1076003)(6636002)(6486002)(6666004)(38100700002)(6506007)(53546011)(2906002)(83380400001)(82960400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pxv+62tVCFxJmMHvygFSreMUjDzeS0utDnXYJyBBhBD32M7RsCub+VcLBxsX?= =?us-ascii?Q?LJDidY+752rcYz9TBxL8g8n5/5EgwevEEg0t176rFhluFzhf3uWLiBndEQvZ?= =?us-ascii?Q?wQ7OexqXknh9qJwfk/XC1EPI9lck7DC9tFgUjYEZTyHkQhZUx8VXW+45QAcd?= =?us-ascii?Q?nI88BT1MhcEMUkf32cRie3nJSuu1fyOAQ2SDXt6zWbijqEP3KsjSNwkASyxZ?= =?us-ascii?Q?j//IZsq44gk8jx5mVUl5VY11dTI9uRZtSD3s34GSJrNol3MsT8JkFICuDAHP?= =?us-ascii?Q?J3PslEaNdsDviIWTo/+u+8PS3bzyseKTeztCVLOSseUCidWblsDmWAlHt415?= =?us-ascii?Q?BUpFaeGHU4mv2qteDJPIpzGl8vZdMm4DOSYErmhQlJYykn6/3H66yy8EknMO?= =?us-ascii?Q?68LygPhbj+noJi4uWhIwsLtBBkBSzurBP31w2pwkvXTlm5MaA8d8RNnkvmWM?= =?us-ascii?Q?CkZLxZbH6o0IPYpvBb9dzhQw+cJbHMy6rKEegj9d4os3zdgiA3LCO1+KyWZL?= =?us-ascii?Q?veepJPe0msTWtqzTcczlJMVarNfbfjEwKBoBZL3OOC1ILkpcyBE7qvulR3jw?= =?us-ascii?Q?4VfmK+yEHoLL1NBGukgpHiP+/7U2oPiaRgZQ/LK+6ZSpRDX5QDjovkpet66H?= =?us-ascii?Q?b6OdrPRv3B0BmNxJZIeoZnMr7Unz/2B/todi6XZKiXy7Zj4383/7+2EoxgiG?= =?us-ascii?Q?kP/9dty9C4WmoFVaJsBPg1Xl18Y5dDRUa7gTU2k0Pb2aNmS4Ie3GUCGiQ4jX?= =?us-ascii?Q?5pfWClUQXD6mN+gR4VUgBBCEwPpR4Z0HSRGSLDVZoVY0luJ3z3Q7hHv2WGhU?= =?us-ascii?Q?CMCCvclRHsLc8msgD58mapnbnWKJv+A/Vx76lAt4b8xcqbvIOxP8uIYJpSlV?= =?us-ascii?Q?F3I3ls9ubkd1XQPrD0F7aGp42TRU9qTxDNCBhJ41kZjkiNxHkOO6ZBeY4/4d?= =?us-ascii?Q?9WL+ExTmQkeZvCduAz+28IJJ2BkcNLuneRKfS6NZmJyVrD3132aA5+wKkJMC?= =?us-ascii?Q?ts/TJM2yPEOBvfjSWuVir5OgpdJo3BC/uVft12J/fNVd2ekVosnYcpBgUqFS?= =?us-ascii?Q?uVNGPGAyEtSB4R9PwUB/HsQ9qudyxekCYe7bkpc/fkXeafAW6wf2CUwxE7cG?= =?us-ascii?Q?DQmjCa+WGRez8e+OMLDcRGj1egXfyWmqtDyCsoteMRqxIEiBxI6iY9FqaSlS?= =?us-ascii?Q?9uT8bVbbCfXA+AgTqPNSe3t2/LW3PrtqfiCPOVYMcIcLDrnz5DwrkHTCUA5v?= =?us-ascii?Q?aRaMpHb0sjX5M80tLBIcBCSJFyovgvRdoIF/tQua5CAhJwEQRvcNl5BmbiG0?= =?us-ascii?Q?ozEx52cq3w6OL+ooYwoBg1zMSfdSm/5NZihh6MT6TzC5REXfTBNQHQBCus+N?= =?us-ascii?Q?MW9PWy6QaYukif7wBi4tnTnX48CVBaFTcer3XPFVTWVYyUxXI3iZG7GfdFe6?= =?us-ascii?Q?MvsW6W5h1VUBpqMNG99rReEbnevd30FakBbMIlVmg2xfzanstlMgm613HKXC?= =?us-ascii?Q?dgRFGwlNRDTxv6JX2zKbwCQgnaIub/X5k1qOGzUAJBlTIv7RCoqDjBq1Bagr?= =?us-ascii?Q?xiAoHrI16PZ62AGAPWzyw5FKNbsLWXkzDZefxhaj88iLFM8WVEbY1ZYjvYOZ?= =?us-ascii?Q?desIHihKBW7TqK1zGEUl0RgdISb1B0+9mUd2FXhJ4qoiM+y5JfsTvg9n368m?= =?us-ascii?Q?p39HC+0k3oaNF04mfz162/bnMc8=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 172539ac-0122-437e-b191-08da004a445a X-MS-Exchange-CrossTenant-AuthSource: AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2022 14:53:40.2751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9GwABvDdMoPMnnd1jJJSNTVSO9m+6xy/cZlNCvsFmTkTXnY6pchWYKDMQLCIHkVW0C145i5nEYGj7BaNtzJ4+Vgayc+ro8D8yVvW3wU2SL4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR1001MB1171 X-TUID: B3dXfEt8J++5 Am Mon, 7 Mar 2022 15:41:04 +0100 schrieb Jan Kiszka : > On 07.03.22 15:09, Henning Schild wrote: > > Am Mon, 7 Mar 2022 10:46:34 +0100 > > schrieb Jan Kiszka : > > > >> On 07.03.22 10:04, Henning Schild wrote: > >>> Am Sat, 5 Mar 2022 17:34:50 +0100 > >>> schrieb Jan Kiszka : > >>> > >>>> From: Jan Kiszka > >>>> > >>>> Saves space in case the image contains any sparse files. > >>>> > >>>> Signed-off-by: Jan Kiszka > >>>> --- > >>>> meta/classes/vm-img.bbclass | 6 +++--- > >>>> 1 file changed, 3 insertions(+), 3 deletions(-) > >>>> > >>>> diff --git a/meta/classes/vm-img.bbclass > >>>> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644 > >>>> --- a/meta/classes/vm-img.bbclass > >>>> +++ b/meta/classes/vm-img.bbclass > >>>> @@ -93,15 +93,15 @@ do_create_ova() { > >>>> export VM_UUID=$(uuidgen) > >>>> # create ovf > >>>> cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst > > >>>> ${PP_DEPLOY}/${OVA_NAME}.ovf > >>>> - tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} > >>>> ${OVA_NAME}.ovf > >>>> + tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} > >>>> ${OVA_NAME}.ovf > >>>> # VirtualBox needs here a manifest file. VMware does > >>>> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" > >>>> ]; then > >>> > >>> Not sure that already suggested it but we did not do it. > >>> > >>>> echo > >>>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum > >>>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >> > >>>> ${PP_DEPLOY}/${OVA_NAME}.mf echo > >>>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum > >>>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >> > >>>> ${PP_DEPLOY}/${OVA_NAME}.mf > >>>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C > >>>> ${PP_DEPLOY} ${OVA_NAME}.mf > >>>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C > >>>> ${PP_DEPLOY} ${OVA_NAME}.mf fi > >>>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} > >>>> ${VIRTUAL_MACHINE_IMAGE_FILE} > >>>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} > >>>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO > >>> > >>> Did you get around to testing this on vmware and vbox? Do you > >>> think it would need to be tested? I could to that. > >>> > >> > >> That would be valuable. Just add a sparse file to the image (dd > >> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that > >> bloats the files. And, of course, check if things still boot. > > > > NACK, breaks ova import into both vmware and virtualbox. I compared > > and it is really this patch which breaks both. > > > > Did not check the vmware logs but vbox says > > VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE > > Amazingly primitive format, this OVO... Not sure, the ova is just a tar of the disk image and an xml. That disk image itself can itself be sparse. see VMDK_SUBFORMAT > > > > So this tarball does not support it because its consumers both do > > not. > > OK, thanks for testing. We may then consider whether to take patch 2 > as well as it may surprise constrained consumers. Patch 1 is fine as > it only targets Isar itself. I think p2 can default to -S and maybe make the args configurable. But i think Adriaan might have some generic remarks on the targz-img anyways. I usually use it for nfs so for me the S would add value. If i remember to specify it on unpack ;). Henning > Jan >