From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6785506002995249152 X-Received: by 2002:ac2:58c2:: with SMTP id u2mr11116800lfo.206.1582053894696; Tue, 18 Feb 2020 11:24:54 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:9786:: with SMTP id y6ls3153691lji.4.gmail; Tue, 18 Feb 2020 11:24:54 -0800 (PST) X-Google-Smtp-Source: APXvYqyAFE6xx4IiBzbLnR9eMQ2U54h7a1OvtCUr6bIQLZToy2ZtUu+FsmrURWBUjAb9Q/NyuJPA X-Received: by 2002:a2e:9592:: with SMTP id w18mr13641105ljh.98.1582053894093; Tue, 18 Feb 2020 11:24:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1582053894; cv=none; d=google.com; s=arc-20160816; b=WLHL57mqWbEH4pd5w04LeESYhfItFUNPwSZZS7+uENYZr9ju24eV9YOm5T3/m8iQQ8 cQAHqINhZb/TRjy61bsjESeWvCmp6YTLmxoZMOoJmFA0pu4g/WF/sp27L9rkQ0VcfSqd KZkr/9CSsw8wljC5jcpNh/wOC5hVH3SAW3By2+dx8PSpCluWrg/hG/DU2wtHbDEscmad aeDoRHg4lhoIg4qiDWM7OLhJkBqbO1ZSTw1nOZyk37lVsMlutjfAR0LYf6Awty4E3pyM QZlO7Q0WGr3Yuvy/AX7PF+2KH7RROyN2JdCzGyOWSVGqa630kdOiJ4AAtNEZJzQcZwDB AJmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:to:from:date; bh=YnBrKKzz9GP5lnjXvgBRAv2lS7zMrjyOKBhZDCypiLU=; b=eNEqdu2sV9lvZWqUJgQQ7ayq2FPhYeS7ZsSNTzpVkxRu2KbKNdNIcmSgMdf1scKwN5 E7Tmtd+20Y1jTWCyuws5Fo6TZyGVYDrfUocoTeAS11Af8wq/yue7xING0wv11SOJxres Sq1TTkyYogI6ZYeNXQ+81ACTHsYr0yfz2SJ5HPv68sqswHvzZj+pom6Vrim1RFwAt1DW CS0UiX0LfTEpggN4j0ASyCOM99RDGzM6CuBCy+vvqNZ8uOacM4VjD7Lec31ISETRtCSt I5aVjhESS7R0ZV9FYlV4YfIJGuBZ33587XOgO6j2Lutbtdv053cXHY2U6QqnWBJ35FK4 KzHA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id d8si245841lji.0.2020.02.18.11.24.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Feb 2020 11:24:53 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 01IJOrGa016709 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 18 Feb 2020 20:24:53 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.69.212]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 01IJOq2R005358 for ; Tue, 18 Feb 2020 20:24:52 +0100 Date: Tue, 18 Feb 2020 20:24:52 +0100 From: Henning Schild To: Subject: Re: [PATCH] lib/oe/path: try hardlinking instead of guessing when it might fail Message-ID: <20200218202452.728ac7b4@md1za8fc.ad001.siemens.net> In-Reply-To: <20200124145100.3188783f@md1za8fc.ad001.siemens.net> References: <20200124134847.5052-1-henning.schild@siemens.com> <20200124145100.3188783f@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: nHV8cE+2KYbp Merged upstream as f5571bda8327f927feb23b167ab4594b7d0c95bc Henning On Fri, 24 Jan 2020 14:51:00 +0100 "[ext] Henning Schild" wrote: > This is related to fixing an issue with "--exclude-path" in wic. On > the Isar side i have everything in place and might send it. But we > need that upstream fix ... or some workaround if that does not get > merged. > > Henning > > On Fri, 24 Jan 2020 14:48:47 +0100 > Henning Schild wrote: > > > From: Henning Schild > > > > The comparison of the stat st_dev is not enough to judge whether > > hardlinking will work. One example would be where you try and > > hardlink across two bind-mounts of a directory. The st_dev will be > > the same and the operation will still fail. > > > > Instead of implementing a check to try and figure out hardlink > > support just try hardlinking and fall back to a copy when running > > into an exception. > > > > Signed-off-by: Henning Schild > > --- > > meta/lib/oe/path.py | 24 ++++++++++++++++++------ > > 1 file changed, 18 insertions(+), 6 deletions(-) > > > > diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py > > index fa209b9795..082972457b 100644 > > --- a/meta/lib/oe/path.py > > +++ b/meta/lib/oe/path.py > > @@ -99,7 +99,22 @@ def copyhardlinktree(src, dst): > > if os.path.isdir(src) and not len(os.listdir(src)): > > return > > > > - if (os.stat(src).st_dev == os.stat(dst).st_dev): > > + canhard = False > > + testfile = None > > + for root, dirs, files in os.walk(src): > > + if len(files): > > + testfile = os.path.join(root, files[0]) > > + break > > + > > + if testfile is not None: > > + try: > > + os.link(testfile, os.path.join(dst, 'testfile')) > > + os.unlink(os.path.join(dst, 'testfile')) > > + canhard = True > > + except Exception as e: > > + bb.debug(2, "Hardlink test failed with " + str(e)) > > + > > + if (canhard): > > # Need to copy directories only with tar first since cp > > will error if two # writers try and create a directory at the same > > time cmd = "cd %s; find . -type d -print | tar --xattrs > > --xattrs-include='*' -cf - -S -C %s -p --no-recursion --files-from - > > | tar --xattrs --xattrs-include='*' -xhf - -C %s" % (src, src, dst) > > @@ -121,12 +136,9 @@ def copyhardlinktree(src, dst): def > > copyhardlink(src, dst): """Make a hard link when possible, otherwise > > copy.""" > > - # We need to stat the destination directory as the destination > > file probably > > - # doesn't exist yet. > > - dstdir = os.path.dirname(dst) > > - if os.stat(src).st_dev == os.stat(dstdir).st_dev: > > + try: > > os.link(src, dst) > > - else: > > + except OSError: > > shutil.copy(src, dst) > > > > def remove(path, recurse=True): >