From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 20 Aug 2026 15:42:00 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f63.google.com (mail-wm1-f63.google.com [209.85.128.63]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 67KDfxaM013628 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Aug 2026 15:41:59 +0200 Received: by mail-wm1-f63.google.com with SMTP id 5b1f17b1804b1-495474a5fbcsf23582565e9.1 for ; Thu, 20 Aug 2026 06:41:59 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1787233314; cv=pass; d=google.com; s=arc-20260327; b=RSwEdpT2cUIJS+i875rHMjRYZXyLDrhJXFWLyFY8h9AKfCKEyp10bMu8hoGycFGljw n5raMJZ+lKWQ4NKjsAV7Q8KcEpMyQA3HidTAh+4OgudrX4yxxzDJSXsG+hKysZSEPJj4 X5IVc6+NPA4uCIePiy1HKBHRQ9ZLIXB03QZG5reOCT+BaEz421ndOPWU0Us5EdNeuyaY DXHNfX0NhELdZkZZA6G9lrT/IK6N4oOFwCew494TZ425o1ZqL/oY/oAMHsBZ0aJ63A7f SP1yXFMMDYz02w9WoiS1FaT5jKTfpwP7Zn7ax0PloW2a81NuhQVVIfXwudAkX53EWjSi Dtsw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=fNJdWehtfbEzTNACXbuSUR8mp4eUaKI8F7dAmnGGs3E=; fh=vshuPBwfi1SmzTNfxuNOecRk5xO035BeQjum6qXWV9Q=; b=FMIM8BtZu4Aql0oRGK0ERXoql6fgMkS9y3WaJuksi0uE3XHX+PlPY0Tmxisl2vI4Aq MSJipbMk/2rKSV3VSWi2JzsyZuCOVr26YGf2tX7bO+Apd7Hz1GBHpksl0npB8J/zUfl8 V+nYYZHCJwIQVosw2hHo1IWIzo0ow/H52n8REwatRzSYgEYBxuThOpzZYHOZAfV5d64t xzde5Xa6LmHmaXH1mT925tSwg9Rgey3lgj9RjGTW0joYufQGMNW/P+5IbE4NqdRTKOH+ HSYdiQWkOxsoHQJJZ47R+HJ3kHWXcjPhUOJBYsznjmjlqwgaDqaxLzDpVOxgD9Eswegb o9uw==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1787233314; x=1787838114; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :from:to:cc:subject:date:message-id:reply-to:content-type; bh=fNJdWehtfbEzTNACXbuSUR8mp4eUaKI8F7dAmnGGs3E=; b=ipH9ca3EuYfScZQ4FiOTGEU58QG/YMXOzoxIB3ukMfKH26zftDvnKyeSmVQtselej3 S0ytNVy3UMZlk79d7xWA29mG6BKFSP0CuZ8/VLn2oA0gF1GFQg2jj17yPq4kU+vGYMuL 1k2VqxUSOV15tTdTd5kIfWTdj+JaWiL8zyE0wkgeASOaNex0IDpsepa+5QosClJbNyVs SZeGREeiZS4DLY8aiGGwvQf0G+fjlbvP9kuWxsO8L7GrsAppxRUdrhb8Xo4AWh3Kk0Ba dq+wHzbsp4ueYxKfutfubMiKSWm0ZEUlfstTbINxM8uFyFtpKsjw+h7vYAjvBFVAuli7 uf7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787233314; x=1787838114; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fNJdWehtfbEzTNACXbuSUR8mp4eUaKI8F7dAmnGGs3E=; b=k/bo0hpzkoDCUaY5vYxFANMnBGqWPfLw0KLl+APp8AjlktsUfYAAer/f5qMhSweirW V2SxdbB4rCNHIA1es27lJNIwIEecDsvLqauYGHiJ69+V3D0eVwFjj6khUWqVdjWOG+JY rQPzy6miqqB+LLIdszLaxyb6lOHQVTlT2KySGhGM1BTztwllkxy+1i6QHdyIuZlc9On9 FDEKczjvZSH0O6zKIWCIjgwuYSmPOOvkRBcU/O+Qbs7U/oNQE15jpwDkK1MZuu5QssuX Q8UjZ11lDfDgDwwTfZOkjU0+VQTuXmt+kX90l/8KL4r2sc2E7Sqd1FrOD23KqBitKDOF GIbw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AHgh+Ro/v21BmK0I6RX3ZemvD96yYfCxheEjVrHf4R6KEKdqpa+bM0Nec6eiyZ8bnfsHujhme1xa@ilbers.de X-Gm-Message-State: AOJu0YwYSf62V1b6vQZ1Do451M6Tr/yQsjmp60Eg1T0tJVDsgTM5mxpa AD14tCVLBp4bPtakHq3IgBCJaHdoaAwktC9CpIO+1WRr5EHTjP5r8Uwj X-Received: by 2002:a05:600c:470a:b0:499:a5fc:207e with SMTP id 5b1f17b1804b1-499aa17e2bamr166762695e9.8.1787233313533; Thu, 20 Aug 2026 06:41:53 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="ATskLdcSDBijICgKNf4Ijs+oFlzHivNRGlkvH+b84cyqZZ5NEg==" Received: by 2002:a05:600c:4e50:b0:495:482d:9805 with SMTP id 5b1f17b1804b1-499b05b5bfcls10977315e9.2.-pod-prod-07-eu; Thu, 20 Aug 2026 06:41:51 -0700 (PDT) X-Received: by 2002:a05:600c:6211:b0:499:8ba4:8cae with SMTP id 5b1f17b1804b1-499aa14aecamr237989045e9.1.1787233311314; Thu, 20 Aug 2026 06:41:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1787233311; cv=none; d=google.com; s=arc-20260327; b=Mpp3m8N4cB7XRjw2ilwayChZGuI2bLB6VL7D+UxqTK1hcJPLWvkY2D7/tqRzgfpPIB CDjD7qpu5LII4pzX07Ar6EoMGOaP0s40ciTyL6SHP6b6MRWA9Zu+BQer783hp0sMRh8A BKJ4eAUwNa+Qa3bxQzlLZTLjqh0DcYI06+Ra0+Hend5X3pid4oPCVXyJwrs9AgWzuDs+ CS1+uWJVJKxzqvZuIBU6VMH2zIwqIGfroFyPB/wU7eWCTo5wc9KZ31EE/lp3gZbturkM RVisV71l5NDAk8mvoVSbKLRo3TUGsV6vVbGtdsUAmuXp3QwamaCjQw1I8PR9F76Ps2bn 1buw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=/JtL2fhoMYFIFDWwyKWVDfMR3QY79CvLv8MvqHX0kHY=; fh=8xvFQ9N6ENvazHnylkO2sbgzUV6bRnZQvNiaC5bb+x4=; b=k9Z7lUs+rDy43FvS+7SPCBX1x4nd6rjOZ6YQdB3nFlrfqHA1acOWYyA+Eo/wIZGeHP hqvtDgHDectP4u+E4ErJ1A3/tPzM+p0jUrqJVnn4mMl+pZ0HfLLQQnwdupicrOP1SjML j1ms1CVTIlV/EF7VStF+6bL34KkmojgIDPiucvwrbxBK/AGTWG9GuanbkXhMQel1/ZjD LwY8lDbSjkQqAQdRbpEW35USplNzb34i4or7Wj14S4qIlkGZ47z4CLdPI1zMpfEhZm3m EqthP2btugM2wsXlEpUiWRPzVg/ONMdZqVhkuzlYygfaq4ZxfvEkXgAfZULhxkD8seae al+A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-499a9e70e00si1107395e9.0.2026.08.20.06.41.51 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 20 Aug 2026 06:41:51 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 67KDf8lO013513 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Aug 2026 15:41:44 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Uladzimir Bely , Aliaksei Karpovich , Aliaksei Laurynovich Subject: [v9 01/13] scripts: Add debrepo python script handling base-apt Date: Thu, 20 Aug 2026 15:29:16 +0200 Message-ID: <20260820133934.383231-2-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260820133934.383231-1-akarpovich@ilbers.de> References: <20260820133934.383231-1-akarpovich@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: 0FMcJagv8CL7 From: Uladzimir Bely This is the main utility responsible for prefetching packages into local `base-apt` repo from external Debian mirrors. It uses python-apt module and requires some kind of minimal `rootfs` to work (let's call it "debrepo context"). Once initialized with `--init --workdir=`, it stores the initial configuration in `repo.opts` file inside the context and uses it at futher calls. In future, the logic `debrepo` script implements could be directly implemented inside bitbake classes. Signed-off-by: Uladzimir Bely Signed-off-by: Aliaksei Karpovich Signed-off-by: Aliaksei Laurynovich --- scripts/deb-repo | 610 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 610 insertions(+) create mode 100755 scripts/deb-repo diff --git a/scripts/deb-repo b/scripts/deb-repo new file mode 100755 index 00000000..23de0461 --- /dev/null +++ b/scripts/deb-repo @@ -0,0 +1,610 @@ +#!/usr/bin/env python3 + +""" +# This software is a part of Isar. +# Copyright (C) 2026 ilbers GmbH + +# deb-repo: build Debian-like repo using "python3-apt" library. + +When building the image, Isar downloads required Debian packages from external +mirrors. After build completed, it can pick all downloaded packages from DL_DIR +and build local 'base-apt' Debian-like repo from them. + +This tool allows to download packages and create local repo in advance. So, +Isar just uses this local repository and does not interact with external +mirrors. Such approach makes deb-dl import/export functionality redundant. + +Script `deb-repo` works in so-called "context" directory. It means some +rootfs-like directory with bare minimum of directories/files required for +python3-apt to work. + +Context directory path is passed with "--workdir " command-line option. +On context creating, all passed parameters are stored in the context directory +and picked every time the context is used again. + +1. Repo for building Debian system +``` +deb-repo --workdir=d12 --init locales gnupg +``` +Initialize the context in "d12" directory and create a repository in +"d12/repo/apt" directory sufficient to debootstrap default system +(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg" +with all their dependencies will be available in this repo. + +``` +deb-repo --workdir=d12 docbook-to-man +``` +Adds "docbook-to-man" packages with its dependencies to earlier created repo. + +``` +deb-repo --workdir=d12 --srcmode docbook-to-man +``` +Downloads source package for "docbook-to-man" and adds it to the repo + +2. Repo for building Ubuntu system +``` +deb-repo --init --workdir=uf \ +--distro=ubuntu --codename=focal --arch=arm64 \ +--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \ +--repodir=repo/apt --repodbdir=repo/db \ +--mirror=http://ports.ubuntu.com/ubuntu-ports \ +locales gnupg +``` +Initialize the context in "uf" directory and create a repository in "repo/apt" +directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and +source list are specified by corresponding arguments. Packages "locales" and +" gnupg" with the dependencies will be also placed to the repo. + +``` +deb-repo --workdir=uf gnupg,locales +``` +Add "gnupg" and "locales" packages with their dependencies to earlier created +ubuntu repo. Other parameters (distro, codename, arch) are ommited since they +are picked from the context. + +3. Repo for cross-building Debian system +``` +deb-repo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf +``` +Initialize the context in "d11" directory sufficient to deboostrap Debian +Bullseye (amd64) with foreign "armhf" architecture support + +``` +deb-repo --workdir=d11 gcc +``` +Add "gcc" package (amd64 version) to earlier created repo. + +``` +deb-repo --workdir=d11 --crossbuild gcc +``` +Add "gcc" package (armhf version) to earlier created repo. +""" + +import os +import sys +import fcntl + +import argparse +import shutil +import subprocess +import pickle +import urllib.parse + +import apt_pkg +import apt.progress.base + + +REPREPRO_TIMEOUT = 1200 + + +class DebRepo(object): + class DebRepoCtx(object): + def __init__(self, workdir): + self.distro = "debian" + self.codename = "bullseye" + self.arch = "amd64" + self.mirror = "http://deb.debian.org/debian" + + self.repodir = f"{workdir}/repo/apt" + self.repodbdir = f"{workdir}/repo/db" + + self.crossarch = self.arch + self.compatarch = None + self.keydir = "/etc/apt/trusted.gpg.d" + + def __init__(self, args): + self.workdir = os.path.abspath(args.workdir) + self.ctx = self.DebRepoCtx(self.workdir) + + self.cache = None + self.depcache = None + self.sr = None + self.extrarepo = None + + self.ctx_load() + self.ctx_update(args) + self.ctx_save() + + print( + f"ctx workdir: {self.workdir}\n" + f" distro: {self.ctx.distro}\n" + f" codename: {self.ctx.codename}\n" + f" arch: {self.ctx.arch}\n" + f" mirror: {self.ctx.mirror}\n" + f" repodir: {self.ctx.repodir}\n" + f" repodbdir: {self.ctx.repodbdir}\n" + f" crossarch: {self.ctx.crossarch}\n" + f" compatarch: {self.ctx.compatarch}\n" + f" keydir: {self.ctx.keydir}" + ) + + if args.extrarepo: + self.extrarepo = os.path.abspath(args.extrarepo) + + def ctx_load(self): + ctxfile = f"{self.workdir}/deb-repo.ctx" + + if os.path.isfile(ctxfile): + with open(ctxfile, 'rb') as f: + self.ctx = pickle.load(f) + + def ctx_save(self): + ctxfile = f"{self.workdir}/deb-repo.ctx" + + with open(ctxfile, 'wb') as f: + pickle.dump(self.ctx, f) + + def ctx_update(self, args): + if args.distro: + self.ctx.distro = args.distro + if args.codename: + self.ctx.codename = args.codename + if args.arch: + self.ctx.arch = args.arch + if args.mirror: + self.ctx.mirror = args.mirror + + if args.repodir: + self.ctx.repodir = os.path.abspath(args.repodir) + if args.repodbdir: + self.ctx.repodbdir = os.path.abspath(args.repodbdir) + + if args.crossarch: + self.ctx.crossarch = args.crossarch + if args.compatarch: + self.ctx.compatarch = args.compatarch + if args.keydir: + self.ctx.keydir = args.keydir + + def create_rootfs(self, aptsrcsfile): + os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True) + with open(f"{self.workdir}/var/lib/dpkg/status", "w"): + pass + + os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True) + + srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list" + if aptsrcsfile and os.path.exists(aptsrcsfile): + shutil.copy(aptsrcsfile, srcfile) + else: + with open(srcfile, "w") as f: + repo = f"{self.ctx.mirror} {self.ctx.codename} main" + f.write(f"deb {repo}\n") + f.write(f"deb-src {repo}\n") + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial", + exist_ok=True) + + os.makedirs(f"{self.workdir}/tmp", exist_ok=True) + + def create_repo_dist(self): + conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf" + os.makedirs(conf_dir, exist_ok=True) + if not os.path.exists(f"{conf_dir}/distributions"): + with open(f"{conf_dir}/distributions", "w") as f: + f.write(f"Codename: {self.ctx.codename}\n") + f.write( + "Architectures: " + "i386 armhf arm64 amd64 mipsel riscv64 source\n") + f.write("Components: main\n") + + def apt_config(self, init, crossbuild): + if not init and self.ctx.compatarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch + + if not init and self.ctx.arch != self.ctx.crossarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch + apt_pkg.config["APT::Architectures::"] = self.ctx.arch + + apt_pkg.config.set("APT::Architecture", self.ctx.arch) + + apt_pkg.config.set("Dir", self.workdir) + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}") + apt_pkg.config.set("Dir::State::status", + f"{self.workdir}/var/lib/dpkg/status") + + apt_pkg.config.set("APT::Install-Recommends", "0") + apt_pkg.config.set("APT::Install-Suggests", "0") + + # Use host keys for authentification + apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir) + + # Allow using repositories without keys + apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1") + + def mark_essential(self): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + if pkg.essential: + self.depcache.mark_install(pkg) + + def mark_by_prio(self, priority): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + ver = self.depcache.get_candidate_ver(pkg) + if ver and ver.priority <= priority: + self.depcache.mark_install(pkg) + + def mark_pkg(self, name, crossbuild): + pkgname = name + + if pkgname and (pkgname not in self.cache): + # Try for cross arch + if (pkgname, self.ctx.crossarch) in self.cache: + pkgname += f":{self.ctx.crossarch}" + + if pkgname not in self.cache: + print(f"Error: package '{name}' not found") + return False + + pkg = self.cache[pkgname] + + if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions): + if (pkg.has_provides) and (not pkg.has_versions): + print("pkgname is virtual package, selecting best provide") + # Select first provide + pkg_provide = pkg.provides_list[0][2] + # Find better provide with higher version + for provide in pkg.provides_list: + if apt_pkg.version_compare(provide[2].ver_str, + pkg_provide.ver_str) > 0: + pkg_provide = provide[2] + self.depcache.mark_install(pkg_provide.parent_pkg) + else: + self.depcache.mark_install(pkg) + else: + version = pkg.version_list[0] + if version.arch == "all": + self.depcache.mark_install(pkg) + else: + if version.multi_arch == version.MULTI_ARCH_FOREIGN: + if (pkgname, self.ctx.arch) in self.cache: + nativepkg = self.cache[pkgname, self.ctx.arch] + self.depcache.mark_install(nativepkg) + else: + return False + else: + if (pkgname, self.ctx.crossarch) in self.cache: + crosspkg = self.cache[pkgname, self.ctx.crossarch] + self.depcache.mark_install(crosspkg) + else: + return False + + return True + + def mark_list(self, pkglist, crossbuild): + ret = True + if pkglist: + for pkgname in pkglist: + ret = ret and self.mark_pkg(pkgname, crossbuild) + + return ret + + def handle_deb(self, item): + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "includedeb", + self.ctx.codename, + item.destfile + ], timeout=REPREPRO_TIMEOUT) + fd.close() + + def handle_repo(self, fetcher): + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + fd = open(f"{self.workdir}/{dir_cache}.lock", "w") + fcntl.flock(fd, fcntl.LOCK_EX) + fetcher.run() + fd.close() + for item in fetcher.items: + if item.status == item.STAT_ERROR: + print("Some error ocured: '%s'" % item.error_text) + pass + else: + self.handle_deb(item) + + def get_filename(self, uri): + path = urllib.parse.urlparse(uri).path + unquoted_path = urllib.parse.unquote(path) + basename = os.path.basename(unquoted_path) + return basename + + def fetch_file(self, uri): + filename = self.get_filename(uri) + return subprocess.run([ + "wget", + "-H", + "--timeout=30", + "--tries=3", + "-nv", + uri, + "-O", + f"{self.workdir}/tmp/{filename}" + ], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + + def handle_dsc(self, dsc_file): + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + result = subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "-S", "-", "-P" "source", + "--delete", + "includedsc", + self.ctx.codename, + os.path.realpath(f"{self.workdir}/tmp/{dsc_file}") + ], timeout=REPREPRO_TIMEOUT) + fd.close() + return result.returncode == 0 + + def handle_src_list(self, pkgs): + ret = True + if pkgs: + local_repo = f"{self.ctx.repodir}/{self.ctx.distro}" + fetched_files = [] + for pkg in pkgs: + pkgname = pkg + pkgver = "" + if '=' in pkg: + pkgname = pkg.split("=")[0] + pkgver = pkg.split("=")[1] + + self.sr.restart() + while self.sr.lookup(pkgname): + if pkgver and pkgver != self.sr.version: + continue + + for sr_file in self.sr.files: + uri_file = self.sr.index.archive_uri(sr_file.path) + filename = os.path.basename(sr_file.path) + path = urllib.parse.urlparse(uri_file).path.lstrip('/') + # Strip the distro directory prefix from upstream path + # e.g. "debian/pool/main/h/hello/file.dsc" -> "pool/main/h/hello/file.dsc" + parts = path.split('/', 1) + local_path = parts[1] if len(parts) > 1 else path + local_file = f"{local_repo}/{local_path}" + print(uri_file) + if filename not in fetched_files: + if os.path.exists(local_file): + shutil.copy2(local_file, f"{self.workdir}/tmp/{filename}") + else: + result = self.fetch_file(uri_file) + if result.returncode != 0: + print(f"ERROR: cannot download {uri_file}") + print(f"ERROR: {result.stderr}") + fetched_files.append(filename) + if sr_file.type == 'dsc': + dsc_file = filename + + ret = self.handle_dsc(dsc_file) + break + return ret + + def apt_run(self, init, srcmode, pkgs, dscfile, crossbuild): + apt_pkg.init() + + extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list" + if self.extrarepo: + extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list" + with open(extrarepo_list, "w") as f: + distdir=os.path.join(self.extrarepo, "dists") + if os.path.isdir(distdir): + for dist in os.listdir(distdir): + repodir = os.path.join(distdir,dist) + if os.path.isdir(repodir): + for repo in os.listdir(repodir): + if os.path.isdir(os.path.join(repodir, repo)): + f.write(f"deb file://{self.extrarepo} " + f"{dist} {repo}\n") + + sources = apt_pkg.SourceList() + sources.read_main_list() + + progress = apt.progress.text.AcquireProgress() + + self.cache = apt_pkg.Cache() + if init: + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + + self.sr = apt_pkg.SourceRecords() + + if self.extrarepo: + apt_pkg.config.set("Dir::Etc::SourceList", extrarepo_list) + apt_pkg.config.set("APT::Get::List-Cleanup", "0") + extrarepo_sources = apt_pkg.SourceList() + extrarepo_sources.read_main_list() + self.cache.update(progress, extrarepo_sources) + self.cache = apt_pkg.Cache() + os.remove(extrarepo_list) + + self.depcache = apt_pkg.DepCache(self.cache) + + ret = True + + if init: + self.mark_essential() + # 1(required), 2(important), 3(standard), 4(optional), 5(extra) + self.mark_by_prio(1) + + pkgs = list(filter(None, ','.join(pkgs).split(','))) + if srcmode: + ret = self.handle_src_list(set(pkgs)) + else: + ret = self.mark_list(pkgs, crossbuild) + + if dscfile: + fobj = open(dscfile, "r") + + try: + tagfile = apt_pkg.TagFile(fobj) + while tagfile.step() == 1: + deps = tagfile.section.get("Build-Depends", "") + # Remove extra commas and spaces - apt_pkg.parse_src_depends + # doesnt like lines like ", device-tree-compiler" + deps = ', '.join( + [s.strip() for s in deps.split(',') if s.strip()] + ) + print(f"parsed deps: {deps}") + for item in apt_pkg.parse_src_depends(deps, False): + pkgname = item[0][0] + self.mark_pkg(pkgname, crossbuild) + + finally: + fobj.close() + + if not ret: + sys.exit("Some of requested packages not found") + + if init or not srcmode: + fetcher = apt_pkg.Acquire(progress) + pm = apt_pkg.PackageManager(self.depcache) + + recs = apt_pkg.PackageRecords(self.cache) + pm.get_archives(fetcher, sources, recs) + + self.handle_repo(fetcher) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--init", + default=False, action="store_true", + help="initialize context in WORKDIR") + parser.add_argument( + "--workdir", + type=str, required=True, + help="work directory storing deb-repo context") + parser.add_argument( + "--aptsrcsfile", + type=str, metavar="PATH", + help="sources.list file to use when init") + parser.add_argument( + "--srcmode", + default=False, action="store_true", + help="add source packages instead of debs") + parser.add_argument( + "--repodir", + type=str, metavar="REPO", + help="repository directory") + parser.add_argument( + "--repodbdir", + type=str, metavar="REPODB", + help="repository database directory") + parser.add_argument( + "--extrarepo", + type=str, metavar="REPO", + help="extra repository to consider") + parser.add_argument( + "--mirror", + type=str, + help="use custom distro mirror") + parser.add_argument( + "--distro", + type=str, + help="select distro to use") + parser.add_argument( + "--codename", + type=str, + help="distro codename") + parser.add_argument( + "--arch", + type=str, + help="distro arch") + parser.add_argument( + "--compatarch", + type=str, metavar="ARCH", + help="compat arch to use") + parser.add_argument( + "--crossarch", + type=str, metavar="ARCH", + help="cross-build arch") + parser.add_argument( + "--keydir", + type=str, + help="directory with distro keys") + parser.add_argument( + "--no-check-gpg", + default=False, action="store_true", + help="allow insecure repositories") + parser.add_argument( + "--dscfile", + type=str, metavar="PATH", + help="Debian source file to parse") + parser.add_argument( + "--crossbuild", + default=False, action="store_true", + help="add packages with cross arch") + + parser.add_argument( + "packages", + nargs='*', type=str, + help="space- or comma-separated list of packages to add") + + args = parser.parse_args() + + return args + + +def main(): + args = parse_arguments() + + if not (args.init or args.packages or args.dscfile): + sys.exit("Nothing to do") + + workdir = os.path.abspath(args.workdir) + os.makedirs(workdir, exist_ok=True) + + with open(f"{workdir}/deb-repo.lock", "a") as file: + fcntl.flock(file.fileno(), fcntl.LOCK_EX) + + deb_repo = DebRepo(args) + + if args.init: + deb_repo.create_rootfs(args.aptsrcsfile) + deb_repo.create_repo_dist() + + deb_repo.apt_config(args.init, args.crossbuild) + deb_repo.apt_run(args.init, args.srcmode, args.packages, + args.dscfile, args.crossbuild) + + #Unlock deb-repo context + fcntl.flock(file.fileno(), fcntl.LOCK_UN) + + +if __name__ == "__main__": + main() -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820133934.383231-2-akarpovich%40ilbers.de.