From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7008834192641359872 X-Received: by 2002:a05:6402:289b:: with SMTP id eg27mr11879930edb.25.1631871376910; Fri, 17 Sep 2021 02:36:16 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:230c:: with SMTP id l12ls3093196eja.11.gmail; Fri, 17 Sep 2021 02:36:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwUKRpcRannquJj3aEkv8fodvy6JOBIt3ylIybb4ns58M5t3KO0I2ruAaX702D60nEOq+Jf X-Received: by 2002:a17:906:52c5:: with SMTP id w5mr11075119ejn.567.1631871376029; Fri, 17 Sep 2021 02:36:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1631871376; cv=none; d=google.com; s=arc-20160816; b=mpHAEOo0EhrT8bzT1lBrmc0ToW4WiG+L3DwT5qA8/qRscAo1TKp/Z922hG/yET3fGQ 3UYReYipJjpf6iAaawEEh7EeblnatIX5qnUY1V7y96qfFl/RkpNmf9nkJHHcZfoMOHk4 CLNeiqUC2tr9wKUlkZytmOC5GR3ggKGNtuk8RQwIexK64Z4cDBQTZDSlZN2Ls9+UC/Ew EOBn/RTM66eEIb5PGVoNKb669iCJwBb9kkiIp4wYwgJD3ZpP090Hn1hNzfXBFxXyCkcP pYtCN/eLA/S5mWBIlZ0plAF7dY3PbWftpq5/YDJAQGqd+/FDnILe+f5bp9WpocTVJCGw p3gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:mime-version:user-agent :date:message-id:subject:from:to; bh=VCzK6PoBeYprij+XiZEbR5mIe+OeLmZd8bhjJ3S8vEA=; b=RhRw/CK+6iMfpExx5xsS599H/8XJ7g7A0bRfIZ88fJKCMUtyYHn7247xYlcyYcuVMT y1mv7TLh4y/yjuRAW4z3tjb1I0TProGDk6oicVtEcMUEq+lm1QkZYA1UXdarMC7D2qhn WtNOF3YRBsEoYL+MulKpnEEkEg3SgDd1cEsDZ2MLQlt+rw+K/tXe3IrjsgJgF9Tw89bI cPZT4Ia9GErj6iDy9NAhlS2DrplfAsGiv0MQJzbbVTFRZT9gw0xSA9vrgd3+uO0tsKCQ jAsJYzl0JTaC3VtzuDl8Y2/yyiCHNeS20ZToZGOR0qtjUBvyjXQekAUv9TWm97HSs06y Aghg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id r23si755360edy.3.2021.09.17.02.36.15 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Sep 2021 02:36:15 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [192.168.67.164] (mm-240-58-214-37.mgts.dynamic.pppoe.byfly.by [37.214.58.240] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 18H9aD3d026513 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 17 Sep 2021 11:36:13 +0200 To: isar-users From: Anton Mikanovich Subject: [RFC] Test cases regrouping concept Message-ID: <524df8b3-fd03-510d-6eac-5de1126aca16@ilbers.de> Date: Fri, 17 Sep 2021 12:36:07 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TUID: jDhig6R5wUEJ We are about to migrate on Avocado framework for CI test and now we have a proposal about new test case structure. There are 3 main goals we need to have after test case rearrange: 1. Minimize 'fast' stage of the CI. After adding a lot of new test cases our fast mode is no more really fast. There are no need in testing some special stuff like containerized rootfs or cached base repository in the build supposed to be fast. 2. Cover all machines and distros. There are a lot of configurations not covered by current CI. 3. Create some middle stage in between roughly similar to the current fast CI coverage. So we can split all the test cases into 3 groups: 1) Fast (Smoke) testing - extra-fast test running to check that the new code does not introduce regressions into the core Isar logic 2) Regular testing - fast testing for daily incoming patchsets checking, will also include Fast 3) Full testing - full testing that cover all the machines and distros, will also include Fast and Regular We should move linux kernel from Fast to Regular because it takes too much time. Because Avocado supports tags we also can split any group more precisely to be able to run any part of the group (like fast-cross, regular-native, etc). Here is how it can look like: 1. Fast test  a) Cross Fast test   mc:qemuarm64-bullseye:isar-image-base   mc:qemuamd64-focal:isar-image-base  b) Native Fast test   mc:qemuarm-buster:isar-image-base   mc:rpi-stretch:isar-image-base 2. Regular test  a) Cross Regular test   mc:qemuamd64-stretch:isar-image-base   mc:qemuarm64-bookworm:isar-image-base   mc:qemuarm-focal:isar-image-base   mc:de0-nano-soc-buster:isar-image-base   mc:stm32mp15x-buster:isar-image-base   mc:rpi-stretch:isar-image-base  b) Native Regular test   mc:qemuamd64-bullseye:isar-image-base   mc:qemuarm64-stretch:isar-image-base   mc:qemumipsel-bullseye:isar-image-base   mc:qemui386-buster:isar-image-base   mc:nand-ubi-demo-buster:isar-image-ubi   mc:qemuamd64-buster-tgz:isar-image-base   mc:qemuamd64-buster:isar-initramfs  c) SDK test   mc:qemuarm-bullseye:isar-image-base:do_populate_sdk  d) Image rebuild test   mc:qemuamd64-bullseye:isar-image-base  e) Test wic --exclude-path   mc:qemuamd64-bullseye:isar-image-base  f) Containerized rootfs test   mc:container-amd64-bullseye:isar-image-base  g) Containerized SDK test   mc:container-amd64-bullseye:isar-image-base:do_populate_sdk  h) Repro signed test   mc:qemuamd64-bookworm:isar-image-base   mc:qemuarm64-stretch:isar-image-base  i) Repro unsigned   mc:qemuamd64-bullseye:isar-image-base   mc:qemuarm-buster:isar-image-base 3. Full test  a) Cross Full test   mc:qemuamd64-bullseye:isar-image-base   mc:qemuamd64-buster:isar-image-base   mc:qemuarm64-buster:isar-image-base   mc:qemuarm64-focal:isar-image-base   mc:qemuarm-stretch:isar-image-base   mc:qemuarm-bullseye:isar-image-base   mc:qemuarm-bookworm:isar-image-base   mc:qemumipsel-stretch:isar-image-base   mc:qemumipsel-buster:isar-image-base   mc:qemumipsel-bullseye:isar-image-base   mc:qemumipsel-bookworm:isar-image-base   mc:qemumipsel-focal:isar-image-base   mc:qemui386-buster:isar-image-base   mc:qemui386-bullseye:isar-image-base   mc:qemui386-bookworm:isar-image-base   mc:qemui386-focal:isar-image-base   mc:cipamd64-bullseye:isar-image-base   mc:sifive-fu540-sid-ports:isar-image-base   mc:hikey-stretch:isar-image-base   mc:bananapi-buster:isar-image-base   mc:nanopi-neo-buster:isar-image-base  b) Native Full test   mc:qemuarm-stretch:isar-image-base   mc:qemuarm-bullseye:isar-image-base   mc:qemuarm-bookworm:isar-image-base   mc:qemuarm-focal:isar-image-base   mc:qemuarm64-buster:isar-image-base   mc:qemuarm64-bullseye:isar-image-base   mc:qemuarm64-bookworm:isar-image-base   mc:qemuarm64-focal:isar-image-base   mc:qemumipsel-stretch:isar-image-base   mc:qemumipsel-buster:isar-image-base   mc:qemumipsel-bookworm:isar-image-base   mc:qemumipsel-focal:isar-image-base   mc:qemui386-stretch:isar-image-base   mc:qemui386-bullseye:isar-image-base   mc:qemui386-bookworm:isar-image-base   mc:qemui386-focal:isar-image-base   mc:qemuriscv64-sid-ports:isar-image-base  c) Containerized rootfs test   mc:container-amd64-stretch:isar-image-base   mc:container-amd64-buster:isar-image-base   mc:container-amd64-bookworm:isar-image-base Multiconfigs of one subtest will be executed in one bitbake call. We would like to have a testsuite definition for building and running images. If anyone have more ideas how to improve this concept any comments are welcome. -- Anton Mikanovich Promwad Ltd. External service provider of ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn, Germany +49 (89) 122 67 24-0 Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov