From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6542418157487783936 X-Received: by 10.223.180.87 with SMTP id v23mr160215wrd.12.1523545981298; Thu, 12 Apr 2018 08:13:01 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.189.4 with SMTP id j4ls1168889wrh.1.gmail; Thu, 12 Apr 2018 08:13:00 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/mxadjxrl6Olkk5vo7/fUEnraDlsZaqynyEjjqo/ybEn6uzsUFtHQCSPJYTL2wVQ+h3ID+ X-Received: by 10.223.209.67 with SMTP id b3mr153912wri.16.1523545980814; Thu, 12 Apr 2018 08:13:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523545980; cv=none; d=google.com; s=arc-20160816; b=Vtr5ynBBa95cguMRxV+wHA2zR56+YPgk+a4mJKnQzXkfChAAlOLgyHoUh9doaZWHBo mSDv91D2+9QP7nkFHEwlG8xrm/YihPLo2CQsMHyTwKto1Y3FCG2QCQ0bAshGGUjpAUgR MYQNacxC0rb5Or1A7Owo6Q1qnX8nwmlgL9/zuGpDoOgWdqdll4wGKyKywHqjx50Jwqz+ r5WvZh55ab3vrsNnsZNEUmrPXR5mQbNO+riFkuOW4nzF3gxx4z22Q8X9V3z5vDjzmKgp NcwHmH0obNwsx8UnLkT2yyfwDBMepnh5gejGssy8OtKfOSmIECe3mtqVzxGtLD7nqTPV EdhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=F+2n8fCu0uoJ98w5Rt0mpuVlnCLCb6I6I7EJgAUVz2g=; b=nsikAn6gNP9NxEZwYaam7MODIa35N8rWBJi4+w05mQNeIiNaMZ63kkl2v6lhd0Niq3 dQqFcsxNlUgpnNGLCSv+9ge+I9CbNgKuvXGBjdN3In+WBrLcAI7cNYhav/svQ1U3b/4l GJMX/0CIZ0h52O7wyjQN1VVv6KYksYSBt4pCOKJ7K3hqVKFySlBfrGBg0BkESRdl0PzO ySocCdRVuth6uoH7AmRQA/ynu3e75XtPxdMsXSHwiSmqkXMbwnArLJJlud799bLaMuC7 HdChEiM9SG2QdyoHEK5rbz2PjZ23+yw3nWKwB7EBk0n39n64DWqTQYj/8eCkdIikU3Ue 1egA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id p21si125883wmc.1.2018.04.12.08.13.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Apr 2018 08:13:00 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id w3CFD0TP005149 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 12 Apr 2018 17:13:00 +0200 Received: from [139.25.69.226] (linux-ses-ext02.ppmd.siemens.net [139.25.69.226]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id w3CFD01f028987 for ; Thu, 12 Apr 2018 17:13:00 +0200 Subject: Re: [DISCUSSION] Mirror implementation with isar-bootstrap To: isar-users@googlegroups.com References: <20180412141233.GD6864@yssyq.radix50.net> From: Claudius Heine Message-ID: <0fd2cd66-a689-febc-ebb6-2690fb4ba0fb@siemens.com> Date: Thu, 12 Apr 2018 17:13:00 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <20180412141233.GD6864@yssyq.radix50.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: gqhrilmcqjpF Hi, On 2018-04-12 16:12, Baurzhan Ismagulov wrote: > Hello Claudius, > > On Mon, Apr 09, 2018 at 02:03:49PM +0200, Claudius Heine wrote: >> AFAIK apt doesn't has a native support for mirror selection that could be >> simply adapted to our use-case. Debian itself uses http redirection to find >> the closest remote server (http://httpredir.debian.org/) > > Which we used to have in Isar master and which failed sporadically with some > 4xx errors, so we've reverted to f.d.o. I didn't suggest changing the mirror here, its just a short report about my research in Debian/Ubuntu-native solutions. >> What we want is being able to overwrite official mirror URIs with our own. >> OEs mechanism to deal with this is the PREMIRRORS variable [1]. >> It basically contains a regex and a replace string separated by a whitespace >> where each of those sets is separated by a new line. >> >> We could do something similar like this: >> >> PREMIRRORS="http://ftp.debian.org/ http://internal.mirror/ \n >> ..." > > I usually prefer to define vars in local.conf and sed > 's/##DEBIANMIRROR##/$DEBIANMIRROR/' debian-stretch.list.in > > debian-stretch.list, so that the user can see what is being used. Patching on > the fly is too magic for me :) . That said, since upstream does it in that way > and people are familiar with it, I'm fine with it. Your method would also have the disadvantage of being much less flexible if multiple repositories are used. (Convenient support for multiple repositories is one of my reasons for doing all this debootstrapizations.) >> If this is not the case, then things are getting more difficult. >> My first question is: Can we assume the internal mirrors are 1:1 copies of >> the external ones? > > I think projects would usually be able to satisfy whatever Isar expects. I'd > assume that an internal mirror to use has all files that are needed for an > Isar-based project. (Depending on what you mean by "1:1": It has to contain all > files needed, not all available in Debian, and possibly in other paths, not > necessarily where f.d.o has them -- that is handled by Packages.gz). What I meant is that such a search and replace method might accidentally overwrite Mirrors you didn't want to change. For example if you use Debian stretch and jessie repos on the same image. The reason doing so might be because you want to use specific packages from stretch, but have all other files from jessie. For some reason your internal mirror just provides a mirror of the stretch distribution. Now its difficult to overwrite just that one entry where stretch is used and leave the jessie one in place. I disregarded those scenarios with the current patch, but it should be possible to change it in a later version if the need arise. >> If not then here are some other solutions: >> >> PREMIRRORS="http://ftp.debian.org/ stretch http://internal.mirror/ \n >> ..." >> >> This way we could select additionally the distribution suite. It's just a >> bit more flexible but maybe not enough? We could add components and what not >> here as well. Maybe we also need to split between 'deb' and 'deb-src' >> mirrors? > > If this is needed at this time, my suggestion would be matching the whole > sources.list line or even the file as you suggest below. That said, I'd be > reluctant to do that if that would break the upstream PREMIRRORS syntax as > people know it. No. Matching a whole source list line is to complicated, matching just some parts of it makes things easier. I would call myself reasonable experienced with regexes, but that format and how it is understood is a bit too much: deb [ option1=value1 option2=value2 ] uri suite [component1] [component2] [...] deb-src [ option1=value1 option2=value2 ] uri suite [component1] [component2] [...] If we want full flexibility, then having a "simpler" matching syntax would help: PREMIRRORS += "type:deb option:arch=amd64 src:ftp\.(\S+\.)?debian.org suite:jessie component:main component:contrib trg:internal.server \n" That would then match all of those entries: deb [ arch=amd64 ] ftp.de.debian.org jessie main contrib deb [ arch=amd64 ] ftp.de.debian.org jessie contrib main and not: deb ftp.de.debian.org jessie main contrib deb [ arch=armhf ] ftp.de.debian.org jessie main contrib deb [ arch=amd64 ] ftp.de.debian.org stretch main contrib deb [ arch=amd64 ] ftp.de.debian.org jessie main contrib non-free Doing the same with one regex is much more painful. > > >> With those previous examples PREMIRRORS are operating on the global >> aggregated source list. So maybe specifying which file they needed to be >> applied could be better: >> >> PREMIRRORS="conf/distro/stretch.list http://ftp.debian.org >> http://internal.mirror \n >> ..." >> >> That might limit the effect of those regexes to lower the risk of >> accidentally overwriting anything else. >> >> I currently don't know how flexible we need to design this, so that is why I >> am asking the community. > > If this is not needed at this time, I'd suggest that we look at your initial > implementation and keep it simple, stupid till it becomes necessary. Then we > would have the exact requirements. I agree. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de