[PATCH 1/2] base.bbclass: Enable the addition of epoch in an apt SRC_URI's version.

[PATCH 1/2] base.bbclass: Enable the addition of epoch in an apt SRC_URI's version.

[Amy Fong]

From: Amy Fong <amy.fong@siemens.com>

If the epoch is added in an apt's package version, there will be an
exception thrown as base.bbclass tries to parse the entry as a URI
(it complains about an invalid port). (man deb-version)

   e.g. SRC_URI = "apt://cryptsetup=2:2.6.1-4~deb12u2

Fix: remove apt:// entries in URI validation list.

Signed-off-by: Amy Fong <amy.fong@siemens.com>
---
 meta/classes/base.bbclass | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index b8825bd3..7d4ab49f 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -123,9 +123,10 @@ python() {
     needsrcrev = False
     srcuri = d.getVar('SRC_URI')
     for uri_string in srcuri.split():
-        uri = bb.fetch.URI(uri_string)
-        if uri.scheme in ("svn", "git", "gitsm", "hg", "p4", "repo"):
-            needsrcrev = True
+        if not uri_string.startswith("apt://"):
+            uri = bb.fetch.URI(uri_string)
+            if uri.scheme in ("svn", "git", "gitsm", "hg", "p4", "repo"):
+                needsrcrev = True
 
     if needsrcrev:
         d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250108155703.10376-1-amy.fong.3142%40gmail.com.

[PATCH 2/2] error directory creation

[Amy Fong]

From: Amy Fong <amy.fong@siemens.com>

On a build, create a directory that gets populated
with temp directories of recipes that fail to build.

The variable ERROR_DIR defines this directory, defaults
to ${TOPDIR}/errorLogs

Signed-off-by: Amy Fong <amy.fong@siemens.com>
---
 bitbake/lib/bb/build.py    | 10 ++++++++++
 bitbake/lib/bb/runqueue.py |  6 ++++++
 meta/conf/bitbake.conf     |  2 ++
 3 files changed, 18 insertions(+)

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 44d08f5c..1c8ad032 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -25,6 +25,7 @@ import bb
 import bb.msg
 import bb.process
 import bb.progress
+import shutil
 from io import StringIO
 from bb import data, event, utils
 
@@ -103,6 +104,15 @@ class TaskFailed(TaskBase):
     def __init__(self, task, fn, logfile, metadata, errprinted = False):
         self.errprinted = errprinted
         super(TaskFailed, self).__init__(task, fn, logfile, metadata)
+        d = metadata
+        errorDir = d.getVar('ERROR_DIR')
+        if errorDir is not None:
+            pkg = d.getVar("PF")
+            newtmp = os.path.join(errorDir, pkg)
+            if os.path.exists(newtmp):
+                shutil.rmtree(newtmp)
+            tempdir = d.getVar('T')
+            shutil.copytree(tempdir, newtmp)
 
 class TaskFailedSilent(TaskBase):
     """Task execution failed (silently)"""
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index bc7e1817..a77749c3 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -25,6 +25,7 @@ from multiprocessing import Process
 import shlex
 import pprint
 import time
+import shutil
 
 bblogger = logging.getLogger("BitBake")
 logger = logging.getLogger("BitBake.RunQueue")
@@ -1334,6 +1335,11 @@ class RunQueue:
         self.rqexe = None
         self.worker = {}
         self.fakeworker = {}
+        errorDir = self.cfgData.getVar('ERROR_DIR')
+        if errorDir is not None:
+            if os.path.exists(errorDir):
+                shutil.rmtree(errorDir)
+            os.mkdir(errorDir)
 
     @staticmethod
     def send_pickled_data(worker, data, name):
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index ef408faa..f345b9cb 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -200,3 +200,5 @@ PATCHRESOLVE ?= "noop"
 
 PREFERRED_PROVIDER_bootstrap-host ??= "isar-mmdebstrap-host"
 PREFERRED_PROVIDER_bootstrap-target ??= "isar-mmdebstrap-target"
+
+ERROR_DIR ?= "${TOPDIR}/errorLogs"
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250108155703.10376-2-amy.fong.3142%40gmail.com.

Re: [PATCH 2/2] error directory creation

['Jan Kiszka' via isar-users]

On 08.01.25 16:57, Amy Fong wrote:
> From: Amy Fong <amy.fong@siemens.com>
> 
> On a build, create a directory that gets populated
> with temp directories of recipes that fail to build.
> 
> The variable ERROR_DIR defines this directory, defaults
> to ${TOPDIR}/errorLogs

This is missing a reason why we want this. I'm also not seeing something
similar in OE.

Jan

> 
> Signed-off-by: Amy Fong <amy.fong@siemens.com>
> ---
>  bitbake/lib/bb/build.py    | 10 ++++++++++
>  bitbake/lib/bb/runqueue.py |  6 ++++++
>  meta/conf/bitbake.conf     |  2 ++
>  3 files changed, 18 insertions(+)
> 
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> index 44d08f5c..1c8ad032 100644
> --- a/bitbake/lib/bb/build.py
> +++ b/bitbake/lib/bb/build.py
> @@ -25,6 +25,7 @@ import bb
>  import bb.msg
>  import bb.process
>  import bb.progress
> +import shutil
>  from io import StringIO
>  from bb import data, event, utils
>  
> @@ -103,6 +104,15 @@ class TaskFailed(TaskBase):
>      def __init__(self, task, fn, logfile, metadata, errprinted = False):
>          self.errprinted = errprinted
>          super(TaskFailed, self).__init__(task, fn, logfile, metadata)
> +        d = metadata
> +        errorDir = d.getVar('ERROR_DIR')
> +        if errorDir is not None:
> +            pkg = d.getVar("PF")
> +            newtmp = os.path.join(errorDir, pkg)
> +            if os.path.exists(newtmp):
> +                shutil.rmtree(newtmp)
> +            tempdir = d.getVar('T')
> +            shutil.copytree(tempdir, newtmp)
>  
>  class TaskFailedSilent(TaskBase):
>      """Task execution failed (silently)"""
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index bc7e1817..a77749c3 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -25,6 +25,7 @@ from multiprocessing import Process
>  import shlex
>  import pprint
>  import time
> +import shutil
>  
>  bblogger = logging.getLogger("BitBake")
>  logger = logging.getLogger("BitBake.RunQueue")
> @@ -1334,6 +1335,11 @@ class RunQueue:
>          self.rqexe = None
>          self.worker = {}
>          self.fakeworker = {}
> +        errorDir = self.cfgData.getVar('ERROR_DIR')
> +        if errorDir is not None:
> +            if os.path.exists(errorDir):
> +                shutil.rmtree(errorDir)
> +            os.mkdir(errorDir)
>  
>      @staticmethod
>      def send_pickled_data(worker, data, name):
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index ef408faa..f345b9cb 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -200,3 +200,5 @@ PATCHRESOLVE ?= "noop"
>  
>  PREFERRED_PROVIDER_bootstrap-host ??= "isar-mmdebstrap-host"
>  PREFERRED_PROVIDER_bootstrap-target ??= "isar-mmdebstrap-target"
> +
> +ERROR_DIR ?= "${TOPDIR}/errorLogs"

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7e83ce0d-f08e-4862-a782-f701d1d86455%40siemens.com.

Re: [PATCH 2/2] error directory creation

[Amy Fong]

[-- Attachment #1: Type: text/plain, Size: 3785 bytes --]

The motivation for creating this error logs directory is to facilitate
troubleshooting
build failures -- after experiencing first hand working with developers not
being able
to find error logs...

Amy


On Wed, Jan 8, 2025 at 3:56 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 08.01.25 16:57, Amy Fong wrote:
> > From: Amy Fong <amy.fong@siemens.com>
> >
> > On a build, create a directory that gets populated
> > with temp directories of recipes that fail to build.
> >
> > The variable ERROR_DIR defines this directory, defaults
> > to ${TOPDIR}/errorLogs
>
> This is missing a reason why we want this. I'm also not seeing something
> similar in OE.
>
> Jan
>
> >
> > Signed-off-by: Amy Fong <amy.fong@siemens.com>
> > ---
> >  bitbake/lib/bb/build.py    | 10 ++++++++++
> >  bitbake/lib/bb/runqueue.py |  6 ++++++
> >  meta/conf/bitbake.conf     |  2 ++
> >  3 files changed, 18 insertions(+)
> >
> > diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> > index 44d08f5c..1c8ad032 100644
> > --- a/bitbake/lib/bb/build.py
> > +++ b/bitbake/lib/bb/build.py
> > @@ -25,6 +25,7 @@ import bb
> >  import bb.msg
> >  import bb.process
> >  import bb.progress
> > +import shutil
> >  from io import StringIO
> >  from bb import data, event, utils
> >
> > @@ -103,6 +104,15 @@ class TaskFailed(TaskBase):
> >      def __init__(self, task, fn, logfile, metadata, errprinted = False):
> >          self.errprinted = errprinted
> >          super(TaskFailed, self).__init__(task, fn, logfile, metadata)
> > +        d = metadata
> > +        errorDir = d.getVar('ERROR_DIR')
> > +        if errorDir is not None:
> > +            pkg = d.getVar("PF")
> > +            newtmp = os.path.join(errorDir, pkg)
> > +            if os.path.exists(newtmp):
> > +                shutil.rmtree(newtmp)
> > +            tempdir = d.getVar('T')
> > +            shutil.copytree(tempdir, newtmp)
> >
> >  class TaskFailedSilent(TaskBase):
> >      """Task execution failed (silently)"""
> > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> > index bc7e1817..a77749c3 100644
> > --- a/bitbake/lib/bb/runqueue.py
> > +++ b/bitbake/lib/bb/runqueue.py
> > @@ -25,6 +25,7 @@ from multiprocessing import Process
> >  import shlex
> >  import pprint
> >  import time
> > +import shutil
> >
> >  bblogger = logging.getLogger("BitBake")
> >  logger = logging.getLogger("BitBake.RunQueue")
> > @@ -1334,6 +1335,11 @@ class RunQueue:
> >          self.rqexe = None
> >          self.worker = {}
> >          self.fakeworker = {}
> > +        errorDir = self.cfgData.getVar('ERROR_DIR')
> > +        if errorDir is not None:
> > +            if os.path.exists(errorDir):
> > +                shutil.rmtree(errorDir)
> > +            os.mkdir(errorDir)
> >
> >      @staticmethod
> >      def send_pickled_data(worker, data, name):
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index ef408faa..f345b9cb 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -200,3 +200,5 @@ PATCHRESOLVE ?= "noop"
> >
> >  PREFERRED_PROVIDER_bootstrap-host ??= "isar-mmdebstrap-host"
> >  PREFERRED_PROVIDER_bootstrap-target ??= "isar-mmdebstrap-target"
> > +
> > +ERROR_DIR ?= "${TOPDIR}/errorLogs"
>
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center
>

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/CAMB4A97bSXwq-JyR10LqvNU9BCk_N35H0J6Aw1dZzRapYy-v0w%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 5211 bytes --]

Re: [PATCH 2/2] error directory creation

['Jan Kiszka' via isar-users]

On 09.01.25 04:43, Amy Fong wrote:
> The motivation for creating this error logs directory is to facilitate
> troubleshooting
> build failures -- after experiencing first hand working with developers
> not being able
> to find error logs...

That is rather a documentation topic, I would say. And bitbake even
prints out where it left the logs for a task that failed.

Jan

> 
> On Wed, Jan 8, 2025 at 3:56 PM Jan Kiszka <jan.kiszka@siemens.com
> <mailto:jan.kiszka@siemens.com>> wrote:
> 
>     On 08.01.25 16:57, Amy Fong wrote:
>     > From: Amy Fong <amy.fong@siemens.com <mailto:amy.fong@siemens.com>>
>     >
>     > On a build, create a directory that gets populated
>     > with temp directories of recipes that fail to build.
>     >
>     > The variable ERROR_DIR defines this directory, defaults
>     > to ${TOPDIR}/errorLogs
> 
>     This is missing a reason why we want this. I'm also not seeing something
>     similar in OE.
> 
>     Jan
> 
>     >
>     > Signed-off-by: Amy Fong <amy.fong@siemens.com
>     <mailto:amy.fong@siemens.com>>
>     > ---
>     >  bitbake/lib/bb/build.py    | 10 ++++++++++
>     >  bitbake/lib/bb/runqueue.py |  6 ++++++
>     >  meta/conf/bitbake.conf     |  2 ++
>     >  3 files changed, 18 insertions(+)
>     >
>     > diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
>     > index 44d08f5c..1c8ad032 100644
>     > --- a/bitbake/lib/bb/build.py
>     > +++ b/bitbake/lib/bb/build.py
>     > @@ -25,6 +25,7 @@ import bb
>     >  import bb.msg
>     >  import bb.process
>     >  import bb.progress
>     > +import shutil
>     >  from io import StringIO
>     >  from bb import data, event, utils
>     > 
>     > @@ -103,6 +104,15 @@ class TaskFailed(TaskBase):
>     >      def __init__(self, task, fn, logfile, metadata, errprinted =
>     False):
>     >          self.errprinted = errprinted
>     >          super(TaskFailed, self).__init__(task, fn, logfile, metadata)
>     > +        d = metadata
>     > +        errorDir = d.getVar('ERROR_DIR')
>     > +        if errorDir is not None:
>     > +            pkg = d.getVar("PF")
>     > +            newtmp = os.path.join(errorDir, pkg)
>     > +            if os.path.exists(newtmp):
>     > +                shutil.rmtree(newtmp)
>     > +            tempdir = d.getVar('T')
>     > +            shutil.copytree(tempdir, newtmp)
>     > 
>     >  class TaskFailedSilent(TaskBase):
>     >      """Task execution failed (silently)"""
>     > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
>     > index bc7e1817..a77749c3 100644
>     > --- a/bitbake/lib/bb/runqueue.py
>     > +++ b/bitbake/lib/bb/runqueue.py
>     > @@ -25,6 +25,7 @@ from multiprocessing import Process
>     >  import shlex
>     >  import pprint
>     >  import time
>     > +import shutil
>     > 
>     >  bblogger = logging.getLogger("BitBake")
>     >  logger = logging.getLogger("BitBake.RunQueue")
>     > @@ -1334,6 +1335,11 @@ class RunQueue:
>     >          self.rqexe = None
>     >          self.worker = {}
>     >          self.fakeworker = {}
>     > +        errorDir = self.cfgData.getVar('ERROR_DIR')
>     > +        if errorDir is not None:
>     > +            if os.path.exists(errorDir):
>     > +                shutil.rmtree(errorDir)
>     > +            os.mkdir(errorDir)
>     > 
>     >      @staticmethod
>     >      def send_pickled_data(worker, data, name):
>     > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>     > index ef408faa..f345b9cb 100644
>     > --- a/meta/conf/bitbake.conf
>     > +++ b/meta/conf/bitbake.conf
>     > @@ -200,3 +200,5 @@ PATCHRESOLVE ?= "noop"
>     > 
>     >  PREFERRED_PROVIDER_bootstrap-host ??= "isar-mmdebstrap-host"
>     >  PREFERRED_PROVIDER_bootstrap-target ??= "isar-mmdebstrap-target"
>     > +
>     > +ERROR_DIR ?= "${TOPDIR}/errorLogs"
> 
>     -- 
>     Siemens AG, Foundational Technologies
>     Linux Expert Center
> 


-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/2d5a8d6b-1340-436a-b67f-d52b47c6342e%40siemens.com.

Re: [PATCH 1/2] base.bbclass: Enable the addition of epoch in an apt SRC_URI's version.

['Jan Kiszka' via isar-users]

On 08.01.25 16:57, Amy Fong wrote:
> From: Amy Fong <amy.fong@siemens.com>
> 
> If the epoch is added in an apt's package version, there will be an
> exception thrown as base.bbclass tries to parse the entry as a URI
> (it complains about an invalid port). (man deb-version)
> 
>    e.g. SRC_URI = "apt://cryptsetup=2:2.6.1-4~deb12u2
> 
> Fix: remove apt:// entries in URI validation list.
> 
> Signed-off-by: Amy Fong <amy.fong@siemens.com>
> ---
>  meta/classes/base.bbclass | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index b8825bd3..7d4ab49f 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -123,9 +123,10 @@ python() {
>      needsrcrev = False
>      srcuri = d.getVar('SRC_URI')
>      for uri_string in srcuri.split():
> -        uri = bb.fetch.URI(uri_string)
> -        if uri.scheme in ("svn", "git", "gitsm", "hg", "p4", "repo"):
> -            needsrcrev = True
> +        if not uri_string.startswith("apt://"):
> +            uri = bb.fetch.URI(uri_string)
> +            if uri.scheme in ("svn", "git", "gitsm", "hg", "p4", "repo"):
> +                needsrcrev = True
>  
>      if needsrcrev:
>          d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")

Looks good to me.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/9b103a8f-21ef-4da2-807c-00b4b5c05fcd%40siemens.com.

Re: [PATCH 1/2] base.bbclass: Enable the addition of epoch in an apt SRC_URI's version.

[Uladzimir Bely]

On Wed, 2025-01-08 at 10:57 -0500, Amy Fong wrote:
> From: Amy Fong <amy.fong@siemens.com>
> 
> If the epoch is added in an apt's package version, there will be an
> exception thrown as base.bbclass tries to parse the entry as a URI
> (it complains about an invalid port). (man deb-version)
> 
>    e.g. SRC_URI = "apt://cryptsetup=2:2.6.1-4~deb12u2
> 
> Fix: remove apt:// entries in URI validation list.
> 
> Signed-off-by: Amy Fong <amy.fong@siemens.com>
> ---
>  meta/classes/base.bbclass | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index b8825bd3..7d4ab49f 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -123,9 +123,10 @@ python() {
>      needsrcrev = False
>      srcuri = d.getVar('SRC_URI')
>      for uri_string in srcuri.split():
> -        uri = bb.fetch.URI(uri_string)
> -        if uri.scheme in ("svn", "git", "gitsm", "hg", "p4",
> "repo"):
> -            needsrcrev = True
> +        if not uri_string.startswith("apt://"):
> +            uri = bb.fetch.URI(uri_string)
> +            if uri.scheme in ("svn", "git", "gitsm", "hg", "p4",
> "repo"):
> +                needsrcrev = True
>  
>      if needsrcrev:
>          d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> -- 
> 2.39.5
> 

Patch 1 applied to next, thanks.

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/558d162879810b5c1fe56b7905bcc1fb78e5e367.camel%40ilbers.de.