[PATCH] meta: Add support for FILESEXTRAPATHS

[PATCH] meta: Add support for FILESEXTRAPATHS

[vijaikumar.kanagarajan]

From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>

OE-core provides FILESEXTRAPATHS to extend the search path for
files and patches. This is particularly useful when you want to
add more files or replace existing files using bbappend.

Bring in support for FILESEXTRAPATHS from OE-core.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 bitbake/conf/bitbake.conf |  3 ++-
 meta/classes/base.bbclass | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
index a460df4..9894ed3 100644
--- a/bitbake/conf/bitbake.conf
+++ b/bitbake/conf/bitbake.conf
@@ -24,12 +24,13 @@ DEPENDS = ""
 DEPLOY_DIR = "${TMPDIR}/deploy"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images"
 DL_DIR = "${TMPDIR}/downloads"
-FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
 FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
+FILESEXTRAPATHS ?= "__default:"
 GITDIR = "${DL_DIR}/git"
 IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
 IMAGE_ROOTFS = "${TMPDIR}/rootfs"
 OVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
+FILESOVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
 P = "${PN}-${PV}"
 PERSISTENT_DIR = "${TMPDIR}/cache"
 PF = "${PN}-${PV}-${PR}"
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 8c7b021..0670430 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -19,6 +19,7 @@
 # OTHER DEALINGS IN THE SOFTWARE.
 
 THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
+FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}","${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files","${FILE_DIRNAME}"], d)}"
 
 def get_deb_host_arch():
     import subprocess
@@ -222,3 +223,21 @@ python do_cleanall() {
     except bb.fetch2.BBFetchException as e:
         bb.fatal(str(e))
 }
+
+# Derived from OpenEmbedded Core: meta/classes/utils.bbclass
+def base_set_filespath(path, d):
+    filespath = []
+    extrapaths = (d.getVar("FILESEXTRAPATHS") or "")
+    # Remove default flag which was used for checking
+    extrapaths = extrapaths.replace("__default:", "")
+    # Don't prepend empty strings to the path list
+    if extrapaths != "":
+        path = extrapaths.split(":") + path
+    # The ":" ensures we have an 'empty' override
+    overrides = (":" + (d.getVar("FILESOVERRIDES") or "")).split(":")
+    overrides.reverse()
+    for o in overrides:
+        for p in path:
+            if p != "":
+                filespath.append(os.path.join(p, o))
+    return ":".join(filespath)
-- 
2.17.1

Re: [PATCH] meta: Add support for FILESEXTRAPATHS

[Jan Kiszka]

On 03.10.19 15:54, vijaikumar.kanagarajan@gmail.com wrote:
> From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> 
> OE-core provides FILESEXTRAPATHS to extend the search path for
> files and patches. This is particularly useful when you want to
> add more files or replace existing files using bbappend.
> 
> Bring in support for FILESEXTRAPATHS from OE-core.

Technically, this is not needed, prepending FILESPATH directly also works. It 
might still be a good idea to add this for conformance with OE.

But as this is no a bitbake interface, it should be documented. Also FILESOVERRIDES.

Jan

> 
> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> ---
>   bitbake/conf/bitbake.conf |  3 ++-
>   meta/classes/base.bbclass | 19 +++++++++++++++++++
>   2 files changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
> index a460df4..9894ed3 100644
> --- a/bitbake/conf/bitbake.conf
> +++ b/bitbake/conf/bitbake.conf
> @@ -24,12 +24,13 @@ DEPENDS = ""
>   DEPLOY_DIR = "${TMPDIR}/deploy"
>   DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images"
>   DL_DIR = "${TMPDIR}/downloads"
> -FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
>   FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> +FILESEXTRAPATHS ?= "__default:"
>   GITDIR = "${DL_DIR}/git"
>   IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
>   IMAGE_ROOTFS = "${TMPDIR}/rootfs"
>   OVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
> +FILESOVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
>   P = "${PN}-${PV}"
>   PERSISTENT_DIR = "${TMPDIR}/cache"
>   PF = "${PN}-${PV}-${PR}"
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 8c7b021..0670430 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -19,6 +19,7 @@
>   # OTHER DEALINGS IN THE SOFTWARE.
>   
>   THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
> +FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}","${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files","${FILE_DIRNAME}"], d)}"
>   
>   def get_deb_host_arch():
>       import subprocess
> @@ -222,3 +223,21 @@ python do_cleanall() {
>       except bb.fetch2.BBFetchException as e:
>           bb.fatal(str(e))
>   }
> +
> +# Derived from OpenEmbedded Core: meta/classes/utils.bbclass
> +def base_set_filespath(path, d):
> +    filespath = []
> +    extrapaths = (d.getVar("FILESEXTRAPATHS") or "")
> +    # Remove default flag which was used for checking
> +    extrapaths = extrapaths.replace("__default:", "")
> +    # Don't prepend empty strings to the path list
> +    if extrapaths != "":
> +        path = extrapaths.split(":") + path
> +    # The ":" ensures we have an 'empty' override
> +    overrides = (":" + (d.getVar("FILESOVERRIDES") or "")).split(":")
> +    overrides.reverse()
> +    for o in overrides:
> +        for p in path:
> +            if p != "":
> +                filespath.append(os.path.join(p, o))
> +    return ":".join(filespath)
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] meta: Add support for FILESEXTRAPATHS

[Vijai Kumar K]

On Fri, Oct 04, 2019 at 09:30:17AM +0200, Jan Kiszka wrote:
> On 03.10.19 15:54, vijaikumar.kanagarajan@gmail.com wrote:
> > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > 
> > OE-core provides FILESEXTRAPATHS to extend the search path for
> > files and patches. This is particularly useful when you want to
> > add more files or replace existing files using bbappend.
> > 
> > Bring in support for FILESEXTRAPATHS from OE-core.
> 
> Technically, this is not needed, prepending FILESPATH directly also works.
> It might still be a good idea to add this for conformance with OE.

Right now, yes, we might not need it. We could use the FILESPATH_prepend like you
suggested. But one case which came to my mind while testing both the solutions is that
you could manipulate the order of FILESPATH with FILESEXTRAPATHS and FILESOVERRIDES.
This was particularly useful when I want to replace a file only for particular distro
or a particular machine.

Thanks,
Vijai Kumar K

> 
> But as this is no a bitbake interface, it should be documented. Also FILESOVERRIDES.
> 
> Jan
> 
> > 
> > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > ---
> >   bitbake/conf/bitbake.conf |  3 ++-
> >   meta/classes/base.bbclass | 19 +++++++++++++++++++
> >   2 files changed, 21 insertions(+), 1 deletion(-)
> > 
> > diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
> > index a460df4..9894ed3 100644
> > --- a/bitbake/conf/bitbake.conf
> > +++ b/bitbake/conf/bitbake.conf
> > @@ -24,12 +24,13 @@ DEPENDS = ""
> >   DEPLOY_DIR = "${TMPDIR}/deploy"
> >   DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images"
> >   DL_DIR = "${TMPDIR}/downloads"
> > -FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
> >   FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> > +FILESEXTRAPATHS ?= "__default:"
> >   GITDIR = "${DL_DIR}/git"
> >   IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
> >   IMAGE_ROOTFS = "${TMPDIR}/rootfs"
> >   OVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
> > +FILESOVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
> >   P = "${PN}-${PV}"
> >   PERSISTENT_DIR = "${TMPDIR}/cache"
> >   PF = "${PN}-${PV}-${PR}"
> > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> > index 8c7b021..0670430 100644
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -19,6 +19,7 @@
> >   # OTHER DEALINGS IN THE SOFTWARE.
> >   THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
> > +FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}","${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files","${FILE_DIRNAME}"], d)}"
> >   def get_deb_host_arch():
> >       import subprocess
> > @@ -222,3 +223,21 @@ python do_cleanall() {
> >       except bb.fetch2.BBFetchException as e:
> >           bb.fatal(str(e))
> >   }
> > +
> > +# Derived from OpenEmbedded Core: meta/classes/utils.bbclass
> > +def base_set_filespath(path, d):
> > +    filespath = []
> > +    extrapaths = (d.getVar("FILESEXTRAPATHS") or "")
> > +    # Remove default flag which was used for checking
> > +    extrapaths = extrapaths.replace("__default:", "")
> > +    # Don't prepend empty strings to the path list
> > +    if extrapaths != "":
> > +        path = extrapaths.split(":") + path
> > +    # The ":" ensures we have an 'empty' override
> > +    overrides = (":" + (d.getVar("FILESOVERRIDES") or "")).split(":")
> > +    overrides.reverse()
> > +    for o in overrides:
> > +        for p in path:
> > +            if p != "":
> > +                filespath.append(os.path.join(p, o))
> > +    return ":".join(filespath)
> > 
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

[PATCH v2] meta: Add support for FILESEXTRAPATHS

[vijaikumar.kanagarajan]

From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>

OE-core provides FILESEXTRAPATHS to extend the search path for
files and patches. This is particularly useful when you want to
add more files or replace existing files using bbappend.

Bring in support for FILESEXTRAPATHS from OE-core.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
Changes in v2:
 - Captured documentation for FILESEXTRAPATHS & FILESOVERRIDES.

 bitbake/conf/bitbake.conf |  3 ++-
 doc/user_manual.md        |  2 ++
 meta/classes/base.bbclass | 19 +++++++++++++++++++
 3 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
index a460df4..9894ed3 100644
--- a/bitbake/conf/bitbake.conf
+++ b/bitbake/conf/bitbake.conf
@@ -24,12 +24,13 @@ DEPENDS = ""
 DEPLOY_DIR = "${TMPDIR}/deploy"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images"
 DL_DIR = "${TMPDIR}/downloads"
-FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
 FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
+FILESEXTRAPATHS ?= "__default:"
 GITDIR = "${DL_DIR}/git"
 IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
 IMAGE_ROOTFS = "${TMPDIR}/rootfs"
 OVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
+FILESOVERRIDES = "local:${MACHINE}:${TARGET_OS}:${TARGET_ARCH}"
 P = "${PN}-${PV}"
 PERSISTENT_DIR = "${TMPDIR}/cache"
 PF = "${PN}-${PV}-${PR}"
diff --git a/doc/user_manual.md b/doc/user_manual.md
index c2657da..e096c24 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -321,6 +321,8 @@ Some other variables include:
  - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
  - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional.
  - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping
+ - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
+ - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
 
 ---
 
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 8c7b021..0670430 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -19,6 +19,7 @@
 # OTHER DEALINGS IN THE SOFTWARE.
 
 THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
+FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}","${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files","${FILE_DIRNAME}"], d)}"
 
 def get_deb_host_arch():
     import subprocess
@@ -222,3 +223,21 @@ python do_cleanall() {
     except bb.fetch2.BBFetchException as e:
         bb.fatal(str(e))
 }
+
+# Derived from OpenEmbedded Core: meta/classes/utils.bbclass
+def base_set_filespath(path, d):
+    filespath = []
+    extrapaths = (d.getVar("FILESEXTRAPATHS") or "")
+    # Remove default flag which was used for checking
+    extrapaths = extrapaths.replace("__default:", "")
+    # Don't prepend empty strings to the path list
+    if extrapaths != "":
+        path = extrapaths.split(":") + path
+    # The ":" ensures we have an 'empty' override
+    overrides = (":" + (d.getVar("FILESOVERRIDES") or "")).split(":")
+    overrides.reverse()
+    for o in overrides:
+        for p in path:
+            if p != "":
+                filespath.append(os.path.join(p, o))
+    return ":".join(filespath)
-- 
2.17.1

Re: [PATCH v2] meta: Add support for FILESEXTRAPATHS

[Baurzhan Ismagulov]

Hello Vijai Kumar,

On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar.kanagarajan@gmail.com wrote:
> OE-core provides FILESEXTRAPATHS to extend the search path for
> files and patches. This is particularly useful when you want to
> add more files or replace existing files using bbappend.
> 
> Bring in support for FILESEXTRAPATHS from OE-core.

Thanks. What are the possibilities of adding a test case for this?

With kind regards,
Baurzhan.

Re: [PATCH v2] meta: Add support for FILESEXTRAPATHS

[Vijai Kumar K]

On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
> Hello Vijai Kumar,
> 
> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar.kanagarajan@gmail.com wrote:
> > OE-core provides FILESEXTRAPATHS to extend the search path for
> > files and patches. This is particularly useful when you want to
> > add more files or replace existing files using bbappend.
> > 
> > Bring in support for FILESEXTRAPATHS from OE-core.
> 
> Thanks. What are the possibilities of adding a test case for this?


Baurzhan,

Sorry for the late reply. I was on vacation.

Is there some test suite in ISAR to test features like these? Just wondering
where would the test cases go.

Thanks,
Vijai Kumar K
> 
> With kind regards,
> Baurzhan.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20191007093737.5coyplnujrjobtc4%40yssyq.m.ilbers.de.

Re: [PATCH v2] meta: Add support for FILESEXTRAPATHS

[Jan Kiszka]

On 14.10.19 08:40, Vijai Kumar K wrote:
> On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
>> Hello Vijai Kumar,
>>
>> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar.kanagarajan@gmail.com wrote:
>>> OE-core provides FILESEXTRAPATHS to extend the search path for
>>> files and patches. This is particularly useful when you want to
>>> add more files or replace existing files using bbappend.
>>>
>>> Bring in support for FILESEXTRAPATHS from OE-core.
>>
>> Thanks. What are the possibilities of adding a test case for this?
> 
> 
> Baurzhan,
> 
> Sorry for the late reply. I was on vacation.
> 
> Is there some test suite in ISAR to test features like these? Just wondering
> where would the test cases go.

- add a use case of the new feature a some recipe in meta-isar
- make sure that recipe is built as part of scripts/ci_build.sh, ideally
  the fast variant if the feature is not taking too much extra time

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2] meta: Add support for FILESEXTRAPATHS

[Jan Kiszka]

On 14.10.19 09:49, [ext] Jan Kiszka wrote:
> On 14.10.19 08:40, Vijai Kumar K wrote:
>> On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
>>> Hello Vijai Kumar,
>>>
>>> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar.kanagarajan@gmail.com wrote:
>>>> OE-core provides FILESEXTRAPATHS to extend the search path for
>>>> files and patches. This is particularly useful when you want to
>>>> add more files or replace existing files using bbappend.
>>>>
>>>> Bring in support for FILESEXTRAPATHS from OE-core.
>>>
>>> Thanks. What are the possibilities of adding a test case for this?
>>
>>
>> Baurzhan,
>>
>> Sorry for the late reply. I was on vacation.
>>
>> Is there some test suite in ISAR to test features like these? Just wondering
>> where would the test cases go.
> 
> - add a use case of the new feature a some recipe in meta-isar

                                  ... to some recipe ...

> - make sure that recipe is built as part of scripts/ci_build.sh, ideally
>   the fast variant if the feature is not taking too much extra time
> 
> Jan
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2] meta: Add support for FILESEXTRAPATHS

[Vijai Kumar K]

On Mon, Oct 14, 2019 at 09:49:47AM +0200, Jan Kiszka wrote:
> On 14.10.19 08:40, Vijai Kumar K wrote:
> > On Mon, Oct 07, 2019 at 11:37:38AM +0200, Baurzhan Ismagulov wrote:
> >> Hello Vijai Kumar,
> >>
> >> On Fri, Oct 04, 2019 at 02:15:06PM +0530, vijaikumar.kanagarajan@gmail.com wrote:
> >>> OE-core provides FILESEXTRAPATHS to extend the search path for
> >>> files and patches. This is particularly useful when you want to
> >>> add more files or replace existing files using bbappend.
> >>>
> >>> Bring in support for FILESEXTRAPATHS from OE-core.
> >>
> >> Thanks. What are the possibilities of adding a test case for this?
> > 
> > 
> > Baurzhan,
> > 
> > Sorry for the late reply. I was on vacation.
> > 
> > Is there some test suite in ISAR to test features like these? Just wondering
> > where would the test cases go.
> 
> - add a use case of the new feature a some recipe in meta-isar
> - make sure that recipe is built as part of scripts/ci_build.sh, ideally
>   the fast variant if the feature is not taking too much extra time

I believe we could replace FILESPATH with FILESEXTRAPATHS in the below files.
That should give a test case also would make it easier to override the files
using bbappends. 

meta/recipes-bsp/u-boot/u-boot-custom.inc
meta/recipes-kernel/linux-module/module.inc
meta/recipes-kernel/linux/linux-custom.inc

-FILESPATH =. "${LAYERDIR_core}/recipes-kernel/linux/files:"
+FILESEXTRAPATHS_prepend := "${FILE_DIRNAME}/files:"

Will test and send a v2 with an additional patch for review.

Thanks,
Vijai Kumar K

> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux