From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:ac8:5fd2:: with SMTP id k18mr24783150qta.379.1635251857745; Tue, 26 Oct 2021 05:37:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:620a:4692:: with SMTP id bq18ls7894773qkb.0.gmail; Tue, 26 Oct 2021 05:37:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJygj4leRNVw+1OcF+1X0A1txpGtOu1CEm7afiax2SCoGEFAf+rUypk2qghgReaqPIvwK9St X-Received: by 2002:a05:620a:280d:: with SMTP id f13mr18576146qkp.31.1635251857237; Tue, 26 Oct 2021 05:37:37 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1635251857; cv=pass; d=google.com; s=arc-20160816; b=Y3m/R19RCKGcM9XKqDhDWx6BRqVod0mwMU80Ek6FTXvSsIAOczvFFqMbkPC7scPX9E dUCIqUfJb3gW1Vl4assVMcG2marLCDu+vezdX1s0iHo/iyASZIUS7T2KvYE11nmw5WL3 5evs5JhOdwgAd2iloVmK/9aPDDx0Qv5mbpH9Yn75hxWVkrFGANlK27SJMB4lyL8/U2lh WSKDVZfzx/aMGkRKv0uYundUzeCYHEY6iQSYkJ0GWYSw+EkcoPfqoUGh9PcnENCZNSZR ytJ5Ik+HCo54jqv9R8p909XJsJf4Z+SfzSR+o1RGz+tyFqiceXtRuRmr0T1xPtC+Uz1Y OOeg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:document_confidentiality :msip_labels:content-language:accept-language:in-reply-to:references :message-id:date:thread-index:thread-topic:subject:to:from :dkim-signature; bh=6cEmvPMzeFyL1VfrVOBIn+HYA/7t7vyY/kuh2r5WWi0=; b=vWQdCaZCzyBcf94IXW7+tPSNcBDCNYSBEOanp6lTR7uHeiR7mKbtkpj/tNUDjmLkUh x6pN9xtCShNPZvmO2pJwPduyQJ1XR4qb+3rz8t4QukHL5uinkfMjqLdfsqAjCaYHyxwL soE9e4imzZDnFbpd2JUFbo8aau07yM91+DdZULtaKjhdiNknYKt4P9sflIK3V9flSCan aVXfBKLxQG446p1FS7sgWUq+3rpwZSqXNFDIKSkMzYRwRcItu88V5LaDvt+nrAZABBNM V1qj2nJa0FnRFg7cb+1cppZ/vy/nZQUKG3UTnxu2r39Epqf2rCcQBRIIjK/DWJgEPmCK 1TbA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector1 header.b=dleLGJK4; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 40.107.20.82 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2082.outbound.protection.outlook.com. [40.107.20.82]) by gmr-mx.google.com with ESMTPS id w22si511666qkp.7.2021.10.26.05.37.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Oct 2021 05:37:37 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 40.107.20.82 as permitted sender) client-ip=40.107.20.82; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector1 header.b=dleLGJK4; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 40.107.20.82 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=blHeTpN3KO6Jo8Yct+jVXoYw6HJj1mjjm1zpbVv/HXe+8U+0nFSI3jpLi8z1NBhjq15JQcHP1OFCSYTH9O1mHl0awv8TX79Hz6muEnqAcL4Lmd20BfPxD9gefg1jfYdndvXJiqaUdBecX0keCNQATeIjNpGjJIMmeKSKHWLFNzsTYCtGAzn8sLHmYpJNBHTht+LcF6qb90UKoy/T5z8Np56CMBoWaAqxlz2aShP6KFJaPPSzLiaweGpc9IfcHk+73LGICClRazOK3ej3Lrn0/JywlhqrHCc935gn/ac64JURUu/wU8+HJ9tsXD4UAEOaxbWKXKahvfl7ve28WP7iSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6cEmvPMzeFyL1VfrVOBIn+HYA/7t7vyY/kuh2r5WWi0=; b=fr5LM4qZO/iRm4k1xoX+O73w0xKmM/OL5bGiESqZALGQIOcyBzrc9DAo8pEx3sBnEIt2IQxN/092Kpyb0llDNjxE9HbkAgIxP9mETB9NbyfXJBFw+kokbW5UCm6kr+RM+JUyILJzPa5EspHj/lMEH8gNofzlpoSVcTS4hBM+yTLdGya26CzDoJ/G7WjZdVHci8A0Ep1vyEVarkH4sLr3DTvzjK9kye1YlC84ZYcWbo0HHbbRWprmKAxSAxgXYVDNHc3YIG2TF3n8SusqXImpApbFJRva5x8tRlts6xRkenxuZUPW2fjWF/5Zb6y+4Uh4FK8/IO5jraL0qv2Oljz6KA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6cEmvPMzeFyL1VfrVOBIn+HYA/7t7vyY/kuh2r5WWi0=; b=dleLGJK4mJwuIvBKe/Qo6mRJkvTe+RllpiHEZbYxZmQc/pK0QCjt3s1gVcSOVW9XkdjL2aKTLiFNR7w3iyea+Y0qvFWKUY6SWnfZXGM0uVNNKfB/pPR+ogGZNXKvOqEJEa2w6EGhkpx4YC1NjAb9NlPQAXv5GTWvGbA1FzqhOQw= Received: from AM4PR1001MB1220.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:200:8d::16) by AM9PR10MB4262.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:1f1::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4628.18; Tue, 26 Oct 2021 12:37:35 +0000 Received: from AM4PR1001MB1220.EURPRD10.PROD.OUTLOOK.COM ([fe80::803d:bba5:3561:f6cd]) by AM4PR1001MB1220.EURPRD10.PROD.OUTLOOK.COM ([fe80::803d:bba5:3561:f6cd%6]) with mapi id 15.20.4628.023; Tue, 26 Oct 2021 12:37:35 +0000 From: "Schmidt, Adriaan" To: "Schmidt, Adriaan" , "isar-users@googlegroups.com" Subject: RE: [PATCH v4 00/10] Add sstate-cache Thread-Topic: [PATCH v4 00/10] Add sstate-cache Thread-Index: AQHXymT1RO78iyG+KEqGqA3lwIevyavlNTHA Date: Tue, 26 Oct 2021 12:37:35 +0000 Message-ID: References: <20211026122811.2654125-1-adriaan.schmidt@siemens.com> In-Reply-To: <20211026122811.2654125-1-adriaan.schmidt@siemens.com> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_Enabled=true; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_SetDate=2021-10-26T12:37:33Z; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_Method=Standard; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_Name=restricted-default; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_SiteId=38ae3bcd-9579-4fd4-adda-b42e1495d55a; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_ActionId=05114b0a-b012-409b-8d87-1e192263441c; MSIP_Label_a59b6cd5-d141-4a33-8bf1-0ca04484304f_ContentBits=0 document_confidentiality: Restricted authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0edd42cc-37dc-4071-5595-08d9987d6339 x-ms-traffictypediagnostic: AM9PR10MB4262: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5236; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: P+YQ+0kokzuvnKCKIMfw9kEALsl8mPIkxJ3Oo1gciz8XlIwUTIAEGyN2zv6BCJLdVTHjDu5lyZX8hoKs19vRL2kMOBtaR00uG4qQbp0fCxRYL+hFoM1i2kns4v6jd8VQw5xNMFbD7nXci2b4RFjMp9RaupBZcpDXKiasZr+xDnBDlAJO+/TSpKKlVLPGttZYpFDGa8r+CjkP7kAyyOm4Yn6qC5VjcyfGZQZlsTTlpzWWfz8bqHv48sGl/zfIcGBKJm+1Gl6IAhsd2Unri1S/32ezIZH9CI+hT04zv6qbC6KUtIDyi3pQwtl2HkSlzQA0vGcj63MTKlH4HxlCVTxibDChz5Wx/D0niK5ofxzP/Iirep5e8DFZQhpH0d5sq+sFiNO0bNdG6XZHaGwXJJsSK2Tg9mZ2yaMiq4fPwKjgDPVLU2dH0/EKXagXUWo8td3B+zSAG4yOTnvrp76fmVwNNMl+/P4HbziE0x5H7al71jH7hPcfyE0hEHvwgsQ5RYLi6iIky+3Hesc368G+7l8ypFff90N2AlGiEYEhBwx3y5SW7jSHz5Qahh+cxOpHX55ifP3X73jG5Cr3Zh06qUpuqTj3pRGIPwKmpSPZyDBVp8BkGPos9PKpgBT7af9a5XvBXAGEALp/XIeb5t1+0o9HAkaCIXix/regHOTPhwgrPT/4UAydYJJRee60awVqXQIwL2mxNSHLptqRdv+HRHu/og== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM4PR1001MB1220.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(4636009)(366004)(66476007)(86362001)(26005)(6506007)(53546011)(316002)(186003)(76116006)(33656002)(110136005)(5660300002)(71200400001)(52536014)(8936002)(64756008)(66556008)(66446008)(66946007)(508600001)(8676002)(122000001)(2906002)(38070700005)(38100700002)(83380400001)(82960400001)(9686003)(55016002)(7696005);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?UqG6LD9vxKFFLZF5EtIljQwuXdi9QsEMNWCi6xKVUThME+WFOW7TO4/nP9wh?= =?us-ascii?Q?zGwTLtCX6cBaAbVxfQvouctlytEHlrGsM8gb1BHhLo9j59DMQ/iHnCEwrgie?= =?us-ascii?Q?X9qjfDQGjJrfYEqjudrqIyUGi9WngWlzexNCpBPjNldkQvvhA1KPsmG+6WOC?= =?us-ascii?Q?8WxfD7LDXZKlY7bx/mHr4ooGS5y3cIbmqn3IT2fApjRnxAYbUabGJL8LcgL9?= =?us-ascii?Q?qpKw6YURv1HboMauqEk+su4p9dC3iUpFyjiTx4V6BnY9/jCLBBoZcNwSZtP5?= =?us-ascii?Q?ppbVj1Zbm/3/Xc5VeTx7BEK4eWWgriDLiJ2csWDhU1zBNzjWKeYR6jwKI6nf?= =?us-ascii?Q?FpSZB7/WEu2HAG2Wp34haNou8/BkiSZ2j4vieCT4A38lMnrZ20p9KEx/5CEU?= =?us-ascii?Q?NKdYilJPYye+8UrVKvO1rAsDc5rwBruIYeKq8OdDFbBV0mOdYkvuShj0D1tb?= =?us-ascii?Q?J6dgsddnhLpmuTDhs5afiLMKWq7e4N5NC9BfixzZGS9RU+8qBhHj/pbuSc89?= =?us-ascii?Q?rOsqd3kQQK6zjI6V7326UnqAaeiSzrWa/ks4SvAAbuVTbOSK+3PLz2Lp/6Wh?= =?us-ascii?Q?S1ts7SDjrTS+xxjMJjuOb/AUqXH33eO15JN4tvWw68KqEC0+KL4Vo2eb33i1?= =?us-ascii?Q?i9agq/KKP2tRtWKATijpDImlQbLQgThpLewPwil8mylYCvNx30Hh6WokD+TJ?= =?us-ascii?Q?CUVFd3tMbDNqMiuBPD2paeWv0ofiCRUfXkEQ3y4gkf7JVFqQditHVkxUzs4U?= =?us-ascii?Q?OflmvVAVfv5DGRAffriu3aObaMqcG/W3LMfqousmDYNmq5okLbuXhobdgjhi?= =?us-ascii?Q?EojVscZFsauA3+60HSe/x9rGM05P+aKewhxZbPEM6JyTW1H3Q1ucRWLq/FHl?= =?us-ascii?Q?hhZnHXQRru+FdiSg6F/0hjiHzEj6Py2I56+brvGSodIhIx4WA89F7J+4AtvD?= =?us-ascii?Q?qeZ1TTHFp160rQcwd9XZce5DCth6tTzuj1J3VTajoqpc87CTnK/pApmCn0zp?= =?us-ascii?Q?F9xM+bXfW2mP6b3LSA2mFTYZ9v4pq+sdtqEt2hj4eYPG+qq+i6IqVKwk3DHH?= =?us-ascii?Q?BZ6hoqgt/zDmkPN3S7h3woXqBsRE/wbu8x7GoA1O1AqFr1ezlvaEJT79LWRW?= =?us-ascii?Q?yyRvY9fiafYUkbpL5QK+i46FYvx3JEKfG36Yb5Nb5iGB1Q2q2NGLt/yKzw9U?= =?us-ascii?Q?ss4nRG82N3a5h0aYIFm4s8WnwQ2c1ngqTTnElAGbN6WaSgcssK8eDY/MrV86?= =?us-ascii?Q?H6By2q7KdMCtrvScTjj7pzKYtZeyxp2fct/R266XERqAjBGcD6bRaBlPnQDx?= =?us-ascii?Q?D+LQqXBuzRW7dlTbrL1tD3BJc6lLybkyb3CvSvgDEJdD/wt4+QehL8/kN9Pr?= =?us-ascii?Q?drmjNXKjsT99rj1h40iFKO3yrrlAFdmViwNnsf/v3c/LYplOB3a9SfF6psg3?= =?us-ascii?Q?g7koA5suHO69dIfdaWQ+x8CvQxZvtju4YENkb7ulkXyIlFf1WvhfMDmzRMil?= =?us-ascii?Q?FctJmaeMhrC1lWb8zR7+7SNFM9U7XwcC8KEI7c0Jw7WHXeIcuTQbR69EOrCx?= =?us-ascii?Q?MpuHUAJunL8IDoC+4XBbmXlevF31exZ/siC54y+GHWwYKRqtWGbb/0v1SX+j?= =?us-ascii?Q?ZVjyYEcnOKiuD8UKZj9pHehqpCzYIyg3+nirEdfpxQHWpAi5lkZREGQbhloc?= =?us-ascii?Q?DWNXpQ=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM4PR1001MB1220.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 0edd42cc-37dc-4071-5595-08d9987d6339 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Oct 2021 12:37:35.1662 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: uudVuke/f2EntH3HCcDb1jyg2cgh34kWf1+Ts/59r9wnuN+++WjdZuEVSRFgQJ1dqv+kQtsLu58V96FOePk5+I11H5pxvrhGL5WwHpa03fI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR10MB4262 X-TUID: jawFFo3zTx/g Current status of the sstate patches: - we've been using them internally for a few months without problems - they now pass the "fast-ci" test suite (there were some issues related to= multiconfig) To see the effect of caching, SSTATE_DIR needs to be persistent between bit= bake calls. When running in GitLab CI, I change the script to run the test = suite twice, deleting TMPDIR between the invocations: diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1693a8a..e2fd652 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -27,7 +27,9 @@ fast-ci: except: - schedules script: - scripts/ci_build.sh -q -f -d + - sudo rm -rf build/tmp/ + - scripts/ci_build.sh -q -f -d =20 full-ci: <<: *common-build -- The first run reports: RESULTS : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CA= NCEL 0 JOB TIME : 17724.65 s and the second: RESULTS : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CA= NCEL 0 JOB TIME : 1002.72 s Adriaan > -----Original Message----- > From: Adriaan Schmidt > Sent: Dienstag, 26. Oktober 2021 14:28 > To: isar-users@googlegroups.com > Cc: Schmidt, Adriaan (T RDA IOT SES-DE) > Subject: [PATCH v4 00/10] Add sstate-cache >=20 > This series adds the sstate-cache feature from OE to Isar. The cache hold= s > the results of bootstrapping, rootfs generation (buildchroot, image rootf= s), > and deb package generation. >=20 > To use the cache, the only configuration neccessary is setting SSTATE_DIR= . > The contents of that directory need to be preserved across bitbake > invocations. >=20 > One known weakness is that the package lists of cached rootfs's can run > out of sync with upstream ("apt-get update" only happens at bootstrap tim= e). > But this also happens with an "old" local build dir, and is something tha= t > may be addressed elsewhere. For now, the recommendation is to frequently > clear the cache (in one of our projects we run a nightly "clear&populate > cache" CI job). >=20 > Patches 1..5 fix (unrelated) issues that would otherwise block sstate > caching, > patch 6 copies files from OE, and patches 7..10 add caching to Isar. >=20 > --- > Changes since v3: > - rebase on next > - make sstate understand that images and initramfss are MACHINE specific > (fixes a false-sharing bug that occured while testing with multiconfig) >=20 > Changes since v2: > - rebase on next > - add the explicit isar-apt dependency to all images (not only wic as > previously). > - don't move existing code in isar-bootstrap.inc > - add `do_deploy_deb[deptask] =3D "do_deploy_deb"` to dpkg-base. This is = so > that > packages depend recursively on their dependencies, even when build resu= lts > are taken from cache. > - generally improve commit messages >=20 > Changes since v1: > - fix copy/paste typo in rootfs.bbclass > - add mounting trick to tar rootfs (because --one-file-system > does not stop at bind mounts) > - have install_imager_deps also depend explicitly on isar-apt >=20 > Adriaan Schmidt (10): > oe imports in central location > images: create deploy dir > rootfs: recursively depend on packages > base: remove unneeded "before do_build" task dependencies > dpkg: add explicit dependency to isar-apt > meta: add sstate feature from oe > sstate: configure > sstate: add caching to isar-bootstrap > sstate: add caching to rootfs > sstate: add caching to debian packages >=20 > meta/classes/base.bbclass | 32 +- > meta/classes/container-img.bbclass | 1 + > meta/classes/cpiogz-img.bbclass | 1 + > meta/classes/dpkg-base.bbclass | 34 +- > meta/classes/dpkg.bbclass | 1 + > meta/classes/ext4-img.bbclass | 1 + > meta/classes/fit-img.bbclass | 1 + > meta/classes/image-tools-extension.bbclass | 2 +- > meta/classes/image.bbclass | 3 + > meta/classes/initramfs.bbclass | 3 + > meta/classes/patch.bbclass | 5 - > meta/classes/rootfs.bbclass | 29 +- > meta/classes/sstate.bbclass | 1311 +++++++++++++++++ > meta/classes/targz-img.bbclass | 1 + > meta/classes/ubi-img.bbclass | 1 + > meta/classes/ubifs-img.bbclass | 1 + > meta/classes/wic-img.bbclass | 1 + > meta/conf/bitbake.conf | 10 +- > meta/lib/oe/gpg_sign.py | 130 ++ > meta/lib/oe/sstatesig.py | 603 ++++++++ > .../isar-bootstrap/isar-bootstrap.inc | 24 + > 21 files changed, 2178 insertions(+), 17 deletions(-) > create mode 100644 meta/classes/sstate.bbclass > create mode 100644 meta/lib/oe/gpg_sign.py > create mode 100644 meta/lib/oe/sstatesig.py >=20 > -- > 2.30.2