From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6520131612833742848 X-Received: by 10.25.0.6 with SMTP id 6mr19200lfa.20.1518086453246; Thu, 08 Feb 2018 02:40:53 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.50.16 with SMTP id y16ls416914ljy.6.gmail; Thu, 08 Feb 2018 02:40:52 -0800 (PST) X-Google-Smtp-Source: AH8x226kFVu4tgAl82DpiMnXRh0b6PnHdGs76+qmgiEveF8+84+tqncizIroQhAfk6UJIr+gIhno X-Received: by 10.46.108.23 with SMTP id h23mr14313ljc.19.1518086452710; Thu, 08 Feb 2018 02:40:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518086452; cv=none; d=google.com; s=arc-20160816; b=GfSyQYvncYqWSJ53a+rhYg6yrGRdEr6tH7lpwL9rUF0rTGoz8kab9RzUgHbf++47Py Nc5I13cEpnuDWJJ+xUx3mShQzy6soLPjLS3NpPItqGDny4DMkpCu/CyScZocCy7Qved8 amNEBtA2ThgFIagdzM6dib7ymhcfqRZaZu+DXkRc39gVM4jfLx4snonP87JAADsRHH+B ZNF80eQpw2tFcjtLoVzrM3/YQ/tdN2dJ8vItD8i2FfAcxohJngTNvnyFIX4k5kPnming I//ihWlyH6OhTYI83bTcLYn7gRpWCuvtPPFc1qlaFDX8IGG1fbF01jfW6panol4iWlSl AZgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=GPBw3vBh6S6JUgybFI0FxdhgWl42h5JfJgoVd1yNomA=; b=ZSys1LjnRbFJ+L9EX6LwGSkImz7dFzYYsgUCdd126iBx7RSuxyJY9j8EesMyYnB95z hMiSyBdAeahz7O16YPFkEvS8kOXa0aYmKFK1fRxjpBEfsluFcWKrjHhObsphcCLavCnM b6PSnF56rBY13libV8sGZv5VQMg5tUYWg/MGJ3+L5h3pfxttG5lIkm4+ohG2OzB3WUqs t3Q5rSluCDCQH/WVQrelih510LLGbVNF0c2ELLTm116KsGCeYE1Ky7JmCQdlLs0ma+nY Yn+G+n8xqjFwdzp3G1yj/LYxsWF4/UI0MAX0KicZ6GIsBcer9JDPI48ohnZH0D8q1x9v N/0Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=vJyZNMB6; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::233 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com. [2a00:1450:400c:c09::233]) by gmr-mx.google.com with ESMTPS id o26si187242ljc.5.2018.02.08.02.40.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Feb 2018 02:40:52 -0800 (PST) Received-SPF: pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::233 as permitted sender) client-ip=2a00:1450:400c:c09::233; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=vJyZNMB6; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::233 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: by mail-wm0-x233.google.com with SMTP id r78so9006869wme.0 for ; Thu, 08 Feb 2018 02:40:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GPBw3vBh6S6JUgybFI0FxdhgWl42h5JfJgoVd1yNomA=; b=vJyZNMB6PWUOoIieD4+9ru4Cf9ZkfuNi6SCRNxjUXVM39rtfpOlS4ffa2Mo52FQ+Y2 NRlmxIGeY30Ltj0tubQQoPCKcXIKxD9ZHBbl4qfXQi/WsUJFv6FhHhBDkUguD//IND7A TlE+OVUnNxXuIGHHiT3ICUw8hB5IFCvzGaO9+30zOkeG9H31IoBH2qZ4p63Qlpwtxxye DY6B7Js/vZyLoMNhqwdj6w6Dd3N2Y8e7Ry+uwQwTEQEpMneTPAkSgb0s3/+3mW9G4b3b jfMn288pVQopMlFgFkz0+KMw3sIMMLMdcyIHas68dVorMNMWQx8iHccfUJQ3dipRRD5r rLtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GPBw3vBh6S6JUgybFI0FxdhgWl42h5JfJgoVd1yNomA=; b=s8lBRJ42fdUhqpaQ/5MhtyYYJuHKW0THLamL/dgolQRVApN26pJeMa7sXoNOAka1F9 M5dxUwvMPMMqq89q/zUoEUjcfcuQeDDeXBnhYrwevx4WgTRGOsRXNu9DUx6PsrnZsvfC u2pEJ8R5G/oqu0VjXNE9sMOK5xNI47+5Dv12dKG7ueOmMhCnDSthJT3qHVwQpG/eWdxr UwBBFSLHYVzLMI6ILo4EzMziTe9AtIp6m605rMmdK+LxwUN339cwiwc2xhyq8T32NEfc 5cjbQInWsCXYPW9zhZJHy3ZqRqQ0jNWnmlRsRURMwNHM2sGfa2AYwKVbIElYcDt9TIqT FnTA== X-Gm-Message-State: APf1xPDowFod0ReBnct0+L3dFJybpCOE5wPAX/FirROSyxuNxSpLRNld Oi0n1zaHDsJMSaMjiTNSC61VBw== X-Received: by 10.28.183.8 with SMTP id h8mr637642wmf.72.1518086452104; Thu, 08 Feb 2018 02:40:52 -0800 (PST) Return-Path: Received: from localhost.localdomain (ip-109-41-193-215.web.vodafone.de. [109.41.193.215]) by smtp.gmail.com with ESMTPSA id k74sm764689wrc.2.2018.02.08.02.40.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 08 Feb 2018 02:40:51 -0800 (PST) From: Benedikt Niedermayr X-Google-Original-From: Benedikt Niedermayr To: isar-users@googlegroups.com Cc: Benedikt Niedermayr Subject: [PATCH v2 2/7] Added API class for apt cache. Date: Thu, 8 Feb 2018 11:40:07 +0100 Message-Id: <1518086412-12567-3-git-send-email-Benedikt.Niedermayr@mixed-mode.de> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1518086412-12567-1-git-send-email-Benedikt.Niedermayr@mixed-mode.de> References: <1518086412-12567-1-git-send-email-Benedikt.Niedermayr@mixed-mode.de> X-TUID: H5j/YTRO6GyY This class implements a simple shell based API for accessing the underlying functionality of aptly. It covers following topics: - mirror handling - repo handling - snapshot handling Signed-off-by: Benedikt Niedermayr --- meta/classes/apt-cache.bbclass | 237 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 meta/classes/apt-cache.bbclass diff --git a/meta/classes/apt-cache.bbclass b/meta/classes/apt-cache.bbclass new file mode 100644 index 0000000..18ec80c --- /dev/null +++ b/meta/classes/apt-cache.bbclass @@ -0,0 +1,237 @@ +# This software is a part of ISAR. +# Copyright (C) 2017-2018 Mixed-Mode GmbH +# +# This class implements common functionality for the isar apt cache. + + + +SNAPSHOT_BASENAME="repo-snapshot" +MIRROR_SNAPSHOT_BASENAME="mirror-snapshot" + +ISAR_REPO = "${DEBDISTRONAME}-repo-local" + +ISAR_REPO_PREFIX = "${DEBDISTRONAME}" +ISAR_MIRROR_PREFIX = "${DEBDISTRONAME}-mirror" + +MIRROR = "${DEBDISTRONAME}" +MIRROR_SECURITY = "${DEBDISTRONAME}-security" +MIRROR_UPDATES = "${DEBDISTRONAME}-updates" + +CACHE_DIR = "${TOPDIR}/apt-cache" +CACHE_CONF_DIR = "${CACHE_DIR}/${DISTRO}/conf" +CHROOT_CACHE_DIR = "/apt-cache" +CFG_FILE = "${CACHE_CONF_DIR}/aptly.conf" +ISAR_FIRST_BUILD_DONE = "${CACHE_CONF_DIR}/isar_first_build_done" + +# Reproducible build is turned on per default +REPRODUCIBLE_BUILD_ENABLED ?= "1" + + +# Setup a new empty debian mirror. +# $1: The filter for delimiting packages +# $2: Mirror +# $3: Remote URL +# $4: Suite +mirror_create() { + aptly -config=${CFG_FILE} \ + mirror create \ + -architectures="${DISTRO_ARCH}" \ + -filter="$1" \ + -filter-with-deps \ + $2 $3 $4 +} + + +# Update a mirror, which may result in downloading new packages +# $1: Mirror +mirror_update() { + aptly -config=${CFG_FILE} mirror update $1 +} + + +# Create a snapshot from a given mirror. +# $1: Snapshot name. +# $2: Mirror +snapshot_create_from_mirror() { + aptly -config=${CFG_FILE} snapshot create $1 from mirror $2 +} + + +# Create a snapshot from a given repository database +# $1: Snapshot name. +snapshot_create_from_repo() { + aptly -config=${CFG_FILE} \ + snapshot create \ + $1 from repo \ + ${ISAR_REPO} +} + + +# Rename a snapshot +# $1: Old name. +# $2: New name. +snapshot_rename() { + aptly -config=${CFG_FILE} snapshot rename $1 $2 +} + +# Delete a snapshot +# $1: Snapshot name +snapshot_delete() { + aptly -config=${CFG_FILE} \ + snapshot drop \ + $1 || bbnote "No snapshot to delete" +} + + +# Create the actual repository for a given snapshot. +# $1: Snapshot name. +# $2: Publish prefix +# $3: Distribution +snapshot_publish() { + aptly -config=${CFG_FILE} \ + publish snapshot \ + -architectures="${DISTRO_ARCH}" \ + -distribution="$3" \ + $1 \ + $2 +} + + +# Create a new repository database for Isar generated packages. +# $1: Repository database name +repo_db_create() { + aptly -config=${CFG_FILE} \ + repo create \ + -component="main" \ + -distribution="${DEBDISTRONAME}" \ + $1 +} + +# Create a new repository database, but +# use a snapshot as source. +# $1: Repository database name. +# $2: Snapshot name +repo_db_create_from_snapshot() { + aptly -config=${CFG_FILE} \ + repo create \ + -component="main" \ + -distribution="${DEBDISTRONAME}" \ + $1 \ + from snapshot \ + $2 +} + + +# Drop a created repository database silenty. +repo_db_drop() { + aptly -config=${CFG_FILE} \ + repo drop -force \ + ${ISAR_REPO} || bbnote "No db to drop" + +} + + +# Add packages to the local repository database. +# $1: Repository database name. +# $*: Packages to add. +repo_db_add_package() { + repo=$1 + shift + aptly -config=${CFG_FILE} repo add -force-replace $repo $* +} + + +# Delete packages from the repository database. +# $1: Repository database name. +# $*: Packages to delete. +repo_db_delete_package() { + repo=$1 + shift + aptly -config=${CFG_FILE} repo remove $repo $* +} + + +# Publish repository consumed by apt. This function can only be called +# for repositories +# $1: Repository database name +# $2: Path Prefix +repo_db_publish() { + aptly -config=${CFG_FILE} \ + publish repo \ + -architectures="${DISTRO_ARCH}" \ + $1 \ + $2 + +} + + +# Drop an already published repository. +# $1: Path prefix +repo_db_publish_drop() { + aptly -config=${CFG_FILE} \ + publish drop \ + ${DEBDISTRONAME} \ + $1 || bbnote "Nothing to publish" +} + + +# Switch already published repository to new repository. This +# will change the content of the published repository. This function +# can only be called on published repos. +# $1: Prefix +cache_update_repo() { + aptly -config=${CFG_FILE} \ + publish update \ + -force-overwrite \ + ${DEBDISTRONAME} $1 +} + + +# Add packages to the isar cache. +# $1: Repository database name +# $2: Repository prefix +# $*: Packages to add +cache_add_package() { + repo=$1 + prefix=$2 + shift 2 + repo_db_add_package $repo $* + cache_update_repo $prefix +} + + +# Delete a package from the apt cache. +# This function is used to be called within do_clean(all) task. +# $1: Repository database name +# $2: Repository prefix +# $*: Packages to delete +cache_delete_package() { + repo=$1 + prefix=$2 + shift 2 + repo_db_delete_package $repo $* + cache_update_repo $prefix +} + + +# Create a snapshot, from current repository state +cache_create_repo_snapshot() { + repo_db_publish_drop ${ISAR_REPO_PREFIX} + snapshot_delete ${SNAPSHOT_BASENAME}_${MACHINE} + snapshot_create_from_repo ${SNAPSHOT_BASENAME}_${MACHINE} + repo_db_publish ${ISAR_REPO} ${ISAR_REPO_PREFIX} +} + +# Load the last snapshot and publish it +cache_load_snapshot() { + last_snapshot=$(aptly --config=${CFG_FILE} --raw -sort=time snapshot list | grep "$1" | tail -n 1) + + if [ -z "$last_snapshot" ]; then + bbfatal "Requesting snapshot which has never been created. Repository may be in a inconsistent state. Totally new and clean build required." + fi + bbplain "Loading last snapshot: $last_snapshot" + repo_db_publish_drop ${ISAR_REPO_PREFIX} + repo_db_drop + repo_db_create_from_snapshot ${ISAR_REPO} ${SNAPSHOT_BASENAME}_${MACHINE} + repo_db_publish ${ISAR_REPO} ${ISAR_REPO_PREFIX} +} -- 2.7.4