From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 23 Jan 2026 09:37:29 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f188.google.com (mail-lj1-f188.google.com [209.85.208.188]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 60N8bTNt016505 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 23 Jan 2026 09:37:29 +0100 Received: by mail-lj1-f188.google.com with SMTP id 38308e7fff4ca-38310a79f92sf10594331fa.3 for ; Fri, 23 Jan 2026 00:37:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1769157444; cv=pass; d=google.com; s=arc-20240605; b=KqHy1S7X8odYstmN1euDzmcalzQj9WdV6ZZlEAetAMczw6GvDLanMtyyIIFnJqOI9c XsCt2FTmyuqKZlJFAQVwd4YOxIs6+XllY9XG+oMfiqZ/VT4fdL40WwssEaa8ZQBAPTXk r7vOMYGVIogtZMHUson+BfFVpm9tQXXAMPjycU7pb4dAVA+pqIcIE1zqGEMI1qD2KdTc yXukGP2A3u8l4PT1b+g9ZsyMWOnyu+MgRockh2fJd5OSt0ruNwSO8O6tZ9wb9KttSVib o5GonT/TVOtHA3jpFWxjJ8mv7/zvdua1Wjg38L0jXxNBIkZh945SjnQwIG9H+1FKAcLU gOfA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :sender:dkim-signature; bh=dNwwVutVq3YsRkFBxUsZ09iPLBX8fp1dF7gvhDDv25o=; fh=yTX+gMEDaZdJAdSOsyivOHn0CXmFfHNhAsYNjIS7xvc=; b=kvGJ0rMe25amMrQ2VxR8EcXFfrbkB2hpIE+XarV9KWbf+hScMESV41EZNpAyAKDKdu 6/FvJKWru1iif2mnf338AXtZ7v+RLBqPKSThB/gH4Z7/r4oay8NRhbdFLVTsMH2nhbbn AqBN2muuFkEjK8VsB/HNRcekqk/rSKqy/kvndDzoSn7guZqffFBx2i3qmbcPaaehFvtP 8GhrHy9OkEFjNP1Y+kaIp/HQKlmICvF1q3WsDV6tNdSzjUJka6x9cUsHUvasU9Z1y9+v 4ezFWbeAJLI0GOqwcMqJ7BUfmiAZZNqURkTiX941at6klVqA/qRjWCnuZ9FaD7gdPFty hBAQ==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=wzh@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1769157443; x=1769762243; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:in-reply-to:from:content-language:references:cc :to:subject:user-agent:mime-version:date:message-id:sender:from:to :cc:subject:date:message-id:reply-to; bh=dNwwVutVq3YsRkFBxUsZ09iPLBX8fp1dF7gvhDDv25o=; b=Hjo6lS1dEMr4Puh0EDDANlUljQ8imWLpIKp1ZYnxTiSHokpLPTTbsnV6fJW5ZcMZWq QjVxcun7u7IGUYnOfP7pojDzWYrO7NeHjU3N9ID5v5UMydSQOT16R/Gx2NHBqc+vnOuJ IvfJXVX/NtUgXv9TFcLzvMNzYzv9U8b7BLgXXGIuGo56sOQfJ8x1/VBDMFRoFhMQfiUd Y2wPs3hfWZucDw4Ed9svc7xdKEPIxwz/8z07HzcEkhAfW00aoRnvJt0j4XqXEhkYWlbL TBk7VedLLXI+ZdRHZyqn0zeyAlbSgAoXfJowytt3hmSUUkTpM0hckDSS7IAVc1YjWoo0 rC3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769157444; x=1769762244; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:in-reply-to :from:content-language:references:cc:to:subject:user-agent :mime-version:date:message-id:x-beenthere:x-gm-message-state:sender :from:to:cc:subject:date:message-id:reply-to; bh=dNwwVutVq3YsRkFBxUsZ09iPLBX8fp1dF7gvhDDv25o=; b=BvGPEXs1TDzdEBkue4/iYJFEKmWAaC6QKrRwPIbXWQJVZX6zHDGqfO+DqbzBPZtL3T +536Gj2ickZp+FElTa5RQs8w4ObjtMwYrJAts/W5YNF/FaW3PUs88daEmN/8iRjiTnq7 fuyfjBxUphfzqfuAaAM92X922YpFRy9oZfJW/SIRl1VZWTAgjUixns7LEnFiVRY1ajDl DFPDrsMzOmwPMDviWtxEZdIz95kQPV4t0Xx9LbvJffl8C2JpQ7IJ1mGNUEDk/n+CSvkN 0TRYBl8GlTnkEG1ujXAT+/iZwt4CrTQ+5xZNlicegBwCQAaq7hdWdVddD1rIeK+Idp7c Y7Og== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCURpB0v6pjGuhWlgIXeFmxIjtduC6/QESBQmSFHgSO8Tw3y+6/rXZoUMKibLVwXvUrEj8Cg@ilbers.de X-Gm-Message-State: AOJu0Ywijy0GhhNG7mWJztPKSXNYCtFw5PxBC+5PX1mkD/RAZsftLT48 /tKF4rIbYCFazf85+glZ8IrPQigxfVDRTCOf/wrvjtQB4uKSp+vXe71/ X-Received: by 2002:a05:6512:2393:b0:59d:cf1b:d96c with SMTP id 2adb3069b0e04-59de4a328e2mr644622e87.53.1769157443462; Fri, 23 Jan 2026 00:37:23 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h="AV1CL+Gm/5lK+OniyPuhJWq18iBndm3qNhydx79AinMJaKsFTQ==" Received: by 2002:a05:6512:2352:b0:59b:6cb8:9cf3 with SMTP id 2adb3069b0e04-59dd797aabfls623272e87.1.-pod-prod-03-eu; Fri, 23 Jan 2026 00:37:21 -0800 (PST) X-Received: by 2002:a05:6512:3088:b0:59d:ce04:e9eb with SMTP id 2adb3069b0e04-59de492c6d1mr722516e87.36.1769157440900; Fri, 23 Jan 2026 00:37:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1769157440; cv=none; d=google.com; s=arc-20240605; b=lwXTwH8hzmDHmrXf6JaEI0Y5qrn2obRIY9G1n+ZfuI/0ODFiLeyJ0Xc/FvKNmx5rYj OSDPLoLBc/7nbunosQ9bE7Ce7qLzsvg2U+JOZZDKXI36C4tTexdNQXGv81h4qifhpEL/ +LzE9KXbbQtB7r1IN3RhliKP+Ls7HbNLuPR4wBPhKCHCSkYah8ypZ3AXQ+PuQju+gLgM lLfWwq/nfmeVRdnkKlpNvZJRiuLu2AR1tS3R2XHDts7fxFRG/EldKq8AJFjF8ahj2m+I D7CdMSO7Ca65q61bX67VZ0X+Sd0OQiPd1xoHt589AEU7dXvQrKgPWGV3B2oxmstSCX1b ERkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id; bh=Fv5PYj3H4/o4+vQ9rfKT7mnKGb8T+0KLSOtv5YR5+ag=; fh=C37l5vGm7U2eZ/l7+t9CMsA78Do2Rg4OXXyzPhO3+tA=; b=luz4Zc1eyUusYVriaEhz2LQU94jI6RnbV6cCcxytf7QGxir6bGYyXi53owBW6vDU1Q VDvpvuPjM1zocakUBPdWQbRlf0U1KJfIr6oIIqoMOXccsR/VWaY86SvkOIKUZdHo2fc0 nZtsGT/ga4ifRXOK9j+MUCJ8Nc2gRHkWqMiigJw++y8g0afhuqnPBC5D3jV6/HgHSyr5 q6yiAtLAa9wRzNpS0G6aO4MgenRvHPJFihcEQhI4gfCUSzeySmed9CJOJ8+1RTrZe61c WzjjGCM6wfbdwVs47y7GKqGIWjU6NCY4dNbOXf9AYsEXM17V7QPnIz03FkNV+YQT6B/g 6tcA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=wzh@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-59de490fccesi41368e87.6.2026.01.23.00.37.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 23 Jan 2026 00:37:20 -0800 (PST) Received-SPF: pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from [192.168.178.117] ([88.130.203.42]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 60N8bJMb016490 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Jan 2026 09:37:19 +0100 Message-ID: <27f0e44f-5f2f-4cd2-990d-c232bda4ff38@ilbers.de> Date: Fri, 23 Jan 2026 09:37:19 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 00/20] Various improvements to the testsuite To: isar-users@googlegroups.com, felix.moessbauer@siemens.com, amikan@ilbers.de Cc: cedric.hombourger@siemens.com References: <20260123082501.240751-1-wzh@ilbers.de> Content-Language: en-US From: Zhihang Wei In-Reply-To: <20260123082501.240751-1-wzh@ilbers.de> Content-Type: text/plain; charset="UTF-8"; format=flowed X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: wzh@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of wzh@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=wzh@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: WHsFL9joFdrT On 1/23/26 09:24, Zhihang Wei wrote: > This series aims at simplifying and speeding up the testsuite execution. > This should enable contributors to actually run it and test their patches. > > As pointed out in [1], the testsuite currently tests various features over > and over again, while others are not tested at all. We try to improve this > by focusing on "feature" tests, whereby each test case checks a set of > features instead of a whole image. We further identified long running tasks > (like the kernel builds) and move them to a dedicated test. > > The execution time reduction varies greatly depending on the hardware, > network and sstate usage. When applying these changes on recently added > patches, the time was often decreased by more than 50% on "fast". > > Note, that more refactorings of the testsuite are needed - and planned, > but this series should already address the biggest pain points. > > [1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ > > Changes since v2: > > - move startvm tests after each build to avoid inter-testsuite dependency > - remove SstateTest from full v3 has passed fast and full CI. Zhihang > Changes since v1: > > - continued work on testsuite refactoring, this supersedes: > - testsuite: fix SignatureTest by avoiding absolute path in bblayers > - testsuite: add rootfs target for rootfs only tests > - Enable SState cache in GitLab CI > - testsuite: fix typo in log message in perform_signature_lint > - completed exclusion / refactoring of db2m sbuild chroot > - GitLab CI: increased after script timeout to give sstate upload more time > - GitLab CI: do not upload sstate artifacts of images (as almost no reuse) > - do not add SignatureTest to fast tag (as otherwise VM tests break...) > - rebuild container test images in VM test to not depend on test order > - rebased onto next > > Anton Mikanovich (1): > testsuite: Group prebuilt_containers related test cases together > > MOESSBAUER, Felix (17): > testsuite: move targets with custom kernel to separate test > testsuite: enable ccache on kernel tests > testsuite: make prebuilt container a feature test > testsuite: make compat test standalone test > testsuite: handle IMAGE_INSTALL solely in cibuilder.py > testsuite: limit cross_debsrc test to subset of packages > testsuite: forward SSTATE_MIRRORS into CI env on sstate > ci: add support for sstate cache > testsuite: make test_cross_deps more specific > testsuite: fix typo in log message in perform_signature_lint > testsuite: fix SignatureTest by avoiding absolute path in bblayers > testsuite: use more recent distros in SignatureTest > testsuite: make SignatureTest idempotent > testsuite: add rootfs target for rootfs only tests > testsuite: refactor sbom tests to avoid overhead > testsuite: make sbuild-flavor test standalone > testsuite: skip VM tests if images are not available > > Zhihang Wei (2): > testsuite: Execute startvm testcases after building > testsuite: Remove SstateTest from full testsuite > > .gitlab-ci.yml | 36 +- > meta-test/conf/bblayers.conf.sample | 8 +- > meta-test/conf/local.conf.sample | 10 - > .../hello-isar/hello-isar.bbappend | 3 + > .../recipes-app/libhello/libhello.bbappend | 5 + > .../recipes-core/images/isar-image-ci.bb | 2 - > .../recipes-core/images/isar-rootfs-ci.bb | 17 + > testsuite/cibase.py | 34 +- > testsuite/cibuilder.py | 49 +- > testsuite/citest.py | 740 +++++++++++------- > 10 files changed, 602 insertions(+), 302 deletions(-) > create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend > create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb > -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/27f0e44f-5f2f-4cd2-990d-c232bda4ff38%40ilbers.de.