From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6519381492065370112 X-Received: by 10.223.157.199 with SMTP id q7mr185697wre.11.1517911791472; Tue, 06 Feb 2018 02:09:51 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.142.69 with SMTP id n63ls2032554wrb.11.gmail; Tue, 06 Feb 2018 02:09:51 -0800 (PST) X-Google-Smtp-Source: AH8x226AGL3wiATm6e9UK8CUJC30TmHYeJ1GYr7OQ3NBMLmdh7VzlZXEdSx4wuwRaxqrCsA96hUe X-Received: by 10.223.164.149 with SMTP id g21mr184160wrb.21.1517911791058; Tue, 06 Feb 2018 02:09:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517911791; cv=none; d=google.com; s=arc-20160816; b=rNZhmZNjPq2iRRTWMuZkOZGpzIny6WtA5sLDh8prHybiWWPsbJwJnD1jkASQNY5WpU 3MGdz/YJnN2ssq2y01bxbUdvo1IPYhw+iH0Wia4sm/l05EhEM4Lb6X7b0Y11qgNvZfdM bl0wPeQkoJpTXj9Cw30Yr4CbOt70vMeku/b+/OW17Tt9nWzYCzwCS33TfOoG7zRcCwlc 8dE6YgQP3IgvLwRgX/+Udk1DffDkZWNM/qvgA+adZD8pXl3u0gFARwPYdy/GJYz54Jnk ub75+cX9e8k/MKzf0Y9RtHc0/8hZr/zDhFUBnUTiE5ZB9lq+XJuqiQf1cRfkywZQO04E jg+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=X4bI31NfE3TLeJfPKOoF2mp8qc+9PFcD7yORDCPMGpw=; b=B5iYjSB2cXitig1Ota4qWHKY1NzNuV3NtD+Zi3t9kgkrBopsxIET7CXNHplN1OrWR+ Ynp95LMGCgXBWf+bC2dVBnvkDBX4aHpq8uGVTOwKTLkm8T4wll0FpKmSrPLupRD8FkOF 9GdnpACIMLgJhnf5Lp+GRWQ+zN8ElYDcciMT6QGra1qLd2LXPPiF/FV0DSnrY6Zuiwhk w8yX9YvttSEpwb/jhe+LcV47/mTpHLlgq6ZJsdiM+FypPhuqemFjUXS0fM5UrSB0j2WI eBYjARBDHBDYn5FUyWCMqDBjpyT0FaKmb9Ju/0PjVPd1aObP99iPsGm/andvPZpdmd4R Dg+w== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=u8yzm+I/; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::22f as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com. [2a00:1450:400c:c09::22f]) by gmr-mx.google.com with ESMTPS id f192si564352wmg.1.2018.02.06.02.09.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Feb 2018 02:09:51 -0800 (PST) Received-SPF: pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::22f as permitted sender) client-ip=2a00:1450:400c:c09::22f; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=u8yzm+I/; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::22f as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: by mail-wm0-x22f.google.com with SMTP id b21so2659315wme.4 for ; Tue, 06 Feb 2018 02:09:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=X4bI31NfE3TLeJfPKOoF2mp8qc+9PFcD7yORDCPMGpw=; b=u8yzm+I/pSwrlqgN38hV+sQEszQDRD0PCzyAwDSH/k0IMLtptupbAC+F+W7x+JjgO5 gDHTzxBzYjBCELFVddM77/N8KTT7BPdwRgvfnEG4glKbPs+OnyRQrtoOAyczEdocXIOi w1rbT5wJ+TLW2xkmuA3/wg/0zfvglPU3Uhz8yAh+bs290AdA425EXZ6GGu9k/lU5UlUw ebB8p2ENTzGv9CO82x7yKLQIE7xFmAtkuylbQ5h1kQH5HsSpKhK91MwIEpX2OB1k2SAG Sy3gc+FXOu5ZERC49Zrmw6ey9FJRCDkEIxPPugG31jIgVD7LN/U4M03AXKLFMOYqc/sZ ms6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=X4bI31NfE3TLeJfPKOoF2mp8qc+9PFcD7yORDCPMGpw=; b=XAsMh/nQr8lAAmPzwNsCJMJvG2t52K33fG17JjaZclSkxYU6wJtYJ32Fjgsak0ECia 2QaMwMrX+8RGv3rNBu2gqDp2P90U3JZBpnj7B7Axs4X/rDdX9uBKl7b3CCoTxcGfKANB hSy6WzNoXOoIejAsmdir+aqrrxp6asJ5UugoKz5Vkk89pSu/oyFCn4nRzAPvonEUB8Xs Q7gubvHsgfvAnwV/KYymGT7Pp53+liBC2A9LBVJpLEy66fx1SM91Zr3ZSjLzebxxftDO CeEHSXTDha4Fip1UCAdU2lYoaBNr8K9zMc+lDXtUSs2f9rmboyk7axJvqj3LYfU/jxzH KDPw== X-Gm-Message-State: APf1xPBJhYW7yXY/nh+DAkxMQJ/zgU2qeOh7tzE4r0sN7pMWYa+ExUSE nC/UYiAzvnBiunYQud/g0YLicg== X-Received: by 10.28.241.14 with SMTP id p14mr1330286wmh.20.1517911790404; Tue, 06 Feb 2018 02:09:50 -0800 (PST) Return-Path: Received: from localhost.localdomain (ip-109-41-195-133.web.vodafone.de. [109.41.195.133]) by smtp.gmail.com with ESMTPSA id r5sm13466342wmg.30.2018.02.06.02.09.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 06 Feb 2018 02:09:49 -0800 (PST) From: Benedikt Niedermayr X-Google-Original-From: Benedikt Niedermayr To: isar-users@googlegroups.com Cc: Benedikt Niedermayr Subject: [PATCH 1/6] Implement support for setting up the local apt mirror and isar repository with aptly. Date: Tue, 6 Feb 2018 11:09:34 +0100 Message-Id: <1517911779-30507-2-git-send-email-Benedikt.Niedermayr@mixed-mode.de> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1517911779-30507-1-git-send-email-Benedikt.Niedermayr@mixed-mode.de> References: <1517911779-30507-1-git-send-email-Benedikt.Niedermayr@mixed-mode.de> X-TUID: WTXDbgCWYzkh This patch introduces support for basically setting up two things: - A local debian mirror for mirroring upstream repositories. - A local isar repo, which holds packages created by Isar. The mirror is getting initialized before buildchroot and image rootfs where created. All required packages where extracted from recipes with the get_filter_list() function. This function will run an own parser in order to take all possible append files into account. All packages defined with BUILDCHROOT_PREINSTALL, IMAGE_PREINSTALL and DEBIAN_DEPENDS within Isar where collected and stored in a format aptly understands. Todos: - Creating mirrors for different sections is hardcoded for now. This should be fixed in future since not all distibutions refer to the same remote sections(e.g. raspbian). - Find a clean naming convention for mirrors, repos and snapshots, in order to get multiconfig compatible. Signed-off-by: Benedikt Niedermayr --- meta/recipes-devtools/isar-apt/files/aptly.conf.in | 17 +++ meta/recipes-devtools/isar-apt/isar-apt.bb | 131 +++++++++++++++++---- 2 files changed, 127 insertions(+), 21 deletions(-) create mode 100644 meta/recipes-devtools/isar-apt/files/aptly.conf.in diff --git a/meta/recipes-devtools/isar-apt/files/aptly.conf.in b/meta/recipes-devtools/isar-apt/files/aptly.conf.in new file mode 100644 index 0000000..73e78f1 --- /dev/null +++ b/meta/recipes-devtools/isar-apt/files/aptly.conf.in @@ -0,0 +1,17 @@ +{ + "rootDir": "##CACHE_DIR##", + "downloadConcurrency": 6, + "downloadSpeedLimit": 0, + "architectures": [], + "dependencyFollowSuggests": false, + "dependencyFollowRecommends": true, + "dependencyFollowAllVariants": false, + "dependencyFollowSource": false, + "gpgDisableSign": true, + "gpgDisableVerify": true, + "downloadSourcePackages": false, + "ppaDistributorID": "ubuntu", + "ppaCodename": "", + "S3PublishEndpoints": {}, + "SwiftPublishEndpoints": {} +} diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb index 9c31d12..f1d38bd 100644 --- a/meta/recipes-devtools/isar-apt/isar-apt.bb +++ b/meta/recipes-devtools/isar-apt/isar-apt.bb @@ -1,29 +1,118 @@ -# This software is a part of ISAR. -# Copyright (C) 2015-2017 ilbers GmbH +inherit apt-cache -SRC_URI = "file://distributions.in" +# Setup the apt cache, load snapshot of last build. +do_cache_config() { -CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf" -do_cache_config[dirs] = "${CACHE_CONF_DIR}" -do_cache_config[stamp-extra-info] = "${DISTRO}" -do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock" + # Very first build of isar. Setting up the cache. + if [ ! -e "${CACHE_CONF_DIR}/aptly.conf" ]; then + bbwarn "Setting up local apt mirror" -# Generate reprepro config for current distro if it doesn't exist. Once it's -# generated, this task should do nothing. -do_cache_config() { - if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then - sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ - ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions - fi + # Check if aptly is installed + aptly version || bbfatal "Aptly is not installed. Please install aptly (>= 1.2)" + + sed -e 's|##CACHE_DIR##|${TOPDIR}/apt-cache/${DISTRO}|' \ + ${FILESDIR}/aptly.conf.in > ${CACHE_CONF_DIR}/aptly.conf || \ + rm ${CACHE_CONF_DIR}/aptly.conf + + # Setup Isar repo + repo_db_create ${ISAR_REPO} ${ISAR_REPO_PREFIX} + repo_db_publish ${ISAR_REPO} ${ISAR_REPO_PREFIX} + + # Setup mirrors + pre_fetch="${@get_filter_list(d)}" - path_cache="${DEPLOY_DIR_APT}/${DISTRO}" - path_databases="${DEPLOY_DIR_DB}/${DISTRO}" + mirror_create "$pre_fetch" ${MIRROR} ${DISTRO_APT_SOURCE} ${DISTRO_SUITE} + mirror_create "$pre_fetch" ${MIRROR_SECURITY} ${DISTRO_APT_SOURCE_SEC} ${DISTRO_SUITE}/updates + mirror_create "$pre_fetch" ${MIRROR_UPDATES} ${DISTRO_APT_SOURCE} ${DISTRO_SUITE}-updates - if [ ! -d "${path_databases}" ]; then - reprepro -b ${path_cache} \ - --dbdir ${path_databases} \ - export ${DEBDISTRONAME} + mirror_update ${MIRROR} + mirror_update ${MIRROR_SECURITY} + mirror_update ${MIRROR_UPDATES} + + snapshot_create_from_mirror ${MIRROR_SNAPSHOT_BASENAME} ${MIRROR} + snapshot_create_from_mirror ${MIRROR_SNAPSHOT_BASENAME}-updates ${MIRROR_UPDATES} + snapshot_create_from_mirror ${MIRROR_SNAPSHOT_BASENAME}-security ${MIRROR_SECURITY} + + snapshot_publish ${MIRROR_SNAPSHOT_BASENAME} ${ISAR_MIRROR_PREFIX} ${DISTRO_SUITE} + snapshot_publish ${MIRROR_SNAPSHOT_BASENAME}-updates ${ISAR_MIRROR_PREFIX}-updates ${DISTRO_SUITE} + snapshot_publish ${MIRROR_SNAPSHOT_BASENAME}-security ${ISAR_MIRROR_PREFIX}-security ${DISTRO_SUITE} fi -} + if [ -e "${ISAR_FIRST_BUILD_DONE}" ] && [ "${REPRODUCIBLE_BUILD_ENABLED}" == "1" ]; then + cache_load_snapshot ${SNAPSHOT_BASENAME}_${MACHINE} + fi +} addtask cache_config after do_unpack before do_build +do_cache_config[dirs] += "${CACHE_CONF_DIR}" +do_cache_config[stamp-extra-info] = "${DISTRO}" +do_cache_config[lockfiles] = "${TOPDIR}/apt-cache/isar.lock" + + +# Return a string containing all packages required for setting up the mirror. +# Implementing and running an own parser will also ensure considering bbappend files. +def get_filter_list(d): + import sys + import bb.cooker, bb.cookerdata + + # + # Todo: Move class into lib/oe + # + class DummyConfigParameters(bb.cookerdata.ConfigParameters): + """ Class for generating Dummy config parameters. Required for creating valid + cookerdata.ConfigParameters and setting these within cookerdata.CookerConfiguration + with cookerdata.CookerConfiguration.setConfigParameters(). + """ + def __init__(self, **options): + self.initial_options = options + super(DummyConfigParameters, self).__init__() + + def parseCommandLine(self, argv=sys.argv): + class DummyOptions: + def __init__(self, initial_options): + for key, val in initial_options.items(): + setattr(self, key, val) + + return DummyOptions(self.initial_options), None + + + # Get collections + # A collection object stores information and methods about bbfiles as well as a list + # of bbfiles. + bbfile_config_priorities = [] + collection = bb.cooker.CookerCollectFiles(bbfile_config_priorities) + bbfiles, masked, searchdirs = collection.collect_bbfiles(d, d) + + configparams = DummyConfigParameters() + + # Parse of configuration data for a recipes' environment + configuration = bb.cookerdata.CookerConfiguration() + configuration.setConfigParameters(configparams) + + # Get the databuilder, which is responsible for holding + # config related information. + databuilder = bb.cookerdata.CookerDataBuilder(configuration, False) + databuilder.parseBaseConfiguration() + parser = bb.cache.NoCache(databuilder) + + prefetch_packages = '' + + for bbfile in bbfiles: + appendfiles = collection.get_file_appends(bbfile) + bb.note('Loading data from {}'.format(bbfile)) + data = parser.loadDataFull(bbfile, appendfiles) + + buildchroot_preinstall = data.getVar('BUILDCHROOT_PREINSTALL', True) or '' + image_preinstall = data.getVar('IMAGE_PREINSTALL', True) or '' + debian_depends = data.getVar('DEBIAN_DEPENDS', True) or '' + + buildchroot_preinstall = '|'.join(buildchroot_preinstall.split()) + image_preinstall = '|'.join(image_preinstall.split()) + debian_depends = debian_depends.replace(',', '|') + + if buildchroot_preinstall: + prefetch_packages += buildchroot_preinstall + '|' + if image_preinstall: + prefetch_packages += image_preinstall + '|' + if debian_depends: + prefetch_packages += debian_depends + '|' + return prefetch_packages.strip('|') -- 2.7.4