[PATCH v2 1/2] container_fetcher: Fix missing checksum warning

[PATCH v2 1/2] container_fetcher: Fix missing checksum warning

['Clara Kowalsky' via isar-users]

In case only a tag is specified for a container image in the SRC_URI and
no digest, a warning should be issued with the recommendation to add the
digest of the container image.
So far, we were presenting in the warning the digest of the
architecture-specific image that happened to be fetched first. However,
we actually want to show the multi-arch manifest digest rather than the
architecture-specific one.
In addition, reading the manifest.json does not work at this point
anyway, as skopeo has already packed it into a Docker archive.

Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
 meta/lib/container_fetcher.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
index 0d659154..16467abb 100644
--- a/meta/lib/container_fetcher.py
+++ b/meta/lib/container_fetcher.py
@@ -6,6 +6,7 @@
 import oe.path
 import os
 import tempfile
+import json
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import logger
 from   bb.fetch2 import MissingChecksumEvent
@@ -60,16 +61,17 @@ class Container(FetchMethod):
         if ud.digest:
             return
 
-        checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
-        checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+        inspect_output = runfetchcmd(f"skopeo inspect docker://{ud.container_name}:{ud.tag}", d, True)
+        digest = json.loads(inspect_output)["Digest"]
 
+        checksum_line = f'SRC_URI = "{ud.url};digest={digest}"'
         strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
 
         # If strict checking enabled and neither sum defined, raise error
         if strict == "1":
             raise NoChecksumError(checksum_line)
 
-        checksum_event = {"sha256sum": checksum}
+        checksum_event = {"sha256sum": digest}
         bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
 
         if strict == "ignore":
@@ -77,7 +79,7 @@ class Container(FetchMethod):
 
         # Log missing digest so user can more easily add it
         logger.warning(
-            f"Missing checksum for '{ud.localpath}', consider using this " \
+            f"Missing checksum for '{ud.url}', consider using this " \
             f"SRC_URI in the recipe:\n{checksum_line}")
 
     def unpack(self, ud, rootdir, d):
-- 
2.49.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250626140731.2732545-1-clara.kowalsky%40siemens.com.

[PATCH v2 2/2] container_fetcher: Verify that tag and digest match

['Clara Kowalsky' via isar-users]

If a tag and digest are specified for a container image in the SRC_URI,
the tag is ignored until now and the container image with the matching
digest is fetched.
With this change, the container image is fetched based on the specified
tag and it is checked whether the digest matches. If not, an error is
thrown.

Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
 meta/lib/container_fetcher.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
index 16467abb..08766742 100644
--- a/meta/lib/container_fetcher.py
+++ b/meta/lib/container_fetcher.py
@@ -11,6 +11,7 @@ from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import logger
 from   bb.fetch2 import MissingChecksumEvent
 from   bb.fetch2 import NoChecksumError
+from   bb.fetch2 import ChecksumError
 from   bb.fetch2 import runfetchcmd
 
 class Container(FetchMethod):
@@ -47,6 +48,22 @@ class Container(FetchMethod):
     def download(self, ud, d):
         tarball = ud.localfile[:-len('.zst')]
         with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
+            # If both tag and digest are provided, verify they match
+            if ud.digest and not "tag" in ud.parm:
+                inspect_output = runfetchcmd(f"skopeo inspect docker://{ud.container_name}:{ud.tag}", d, True)
+                actual_digest = json.loads(inspect_output)["Digest"]
+                if actual_digest != ud.digest:
+                    messages = []
+                    messages.append(f"Checksum mismatch for {ud.container_name}:{ud.tag}")
+                    messages.append("If this change is expected (e.g. you have upgraded " \
+                                "to a new version without updating the checksums) " \
+                                "then you can use these lines within the recipe:")
+                    messages.append(f'SRC_URI = "docker://{ud.container_name};digest={actual_digest};tag={ud.tag}"')
+                    messages.append("Otherwise you should retry the download and/or " \
+                                "check with upstream to determine if the container image has " \
+                                "become corrupted or otherwise unexpectedly modified.")
+                    raise ChecksumError("\n".join(messages), ud.url, actual_digest)
+
             # Take a two steps for downloading into a docker archive because
             # not all source may have the required Docker schema 2 manifest.
             runfetchcmd("skopeo copy --preserve-digests " + \
-- 
2.49.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250626140731.2732545-2-clara.kowalsky%40siemens.com.

Re: [PATCH v2 2/2] container_fetcher: Verify that tag and digest match

['Jan Kiszka' via isar-users]

On 26.06.25 16:07, Clara Kowalsky wrote:
> If a tag and digest are specified for a container image in the SRC_URI,
> the tag is ignored until now and the container image with the matching
> digest is fetched.
> With this change, the container image is fetched based on the specified
> tag and it is checked whether the digest matches. If not, an error is
> thrown.
> 
> Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
> ---
>  meta/lib/container_fetcher.py | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
> index 16467abb..08766742 100644
> --- a/meta/lib/container_fetcher.py
> +++ b/meta/lib/container_fetcher.py
> @@ -11,6 +11,7 @@ from   bb.fetch2 import FetchMethod
>  from   bb.fetch2 import logger
>  from   bb.fetch2 import MissingChecksumEvent
>  from   bb.fetch2 import NoChecksumError
> +from   bb.fetch2 import ChecksumError
>  from   bb.fetch2 import runfetchcmd
>  
>  class Container(FetchMethod):
> @@ -47,6 +48,22 @@ class Container(FetchMethod):
>      def download(self, ud, d):
>          tarball = ud.localfile[:-len('.zst')]
>          with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
> +            # If both tag and digest are provided, verify they match
> +            if ud.digest and not "tag" in ud.parm:

Hmm, I'm confused by my own suggestion right now: Did you test that
again? Don't we rather need

if ud.digest and "tag" in ud.parm:

?

> +                inspect_output = runfetchcmd(f"skopeo inspect docker://{ud.container_name}:{ud.tag}", d, True)
> +                actual_digest = json.loads(inspect_output)["Digest"]
> +                if actual_digest != ud.digest:
> +                    messages = []
> +                    messages.append(f"Checksum mismatch for {ud.container_name}:{ud.tag}")
> +                    messages.append("If this change is expected (e.g. you have upgraded " \
> +                                "to a new version without updating the checksums) " \
> +                                "then you can use these lines within the recipe:")
> +                    messages.append(f'SRC_URI = "docker://{ud.container_name};digest={actual_digest};tag={ud.tag}"')
> +                    messages.append("Otherwise you should retry the download and/or " \
> +                                "check with upstream to determine if the container image has " \
> +                                "become corrupted or otherwise unexpectedly modified.")
> +                    raise ChecksumError("\n".join(messages), ud.url, actual_digest)
> +
>              # Take a two steps for downloading into a docker archive because
>              # not all source may have the required Docker schema 2 manifest.
>              runfetchcmd("skopeo copy --preserve-digests " + \

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/bd021656-53cc-4218-9b7d-9ee5323114b9%40siemens.com.

Re: [PATCH v2 2/2] container_fetcher: Verify that tag and digest match

['Clara Kowalsky' via isar-users]

On 26.06.25 18:00, Jan Kiszka wrote:
> On 26.06.25 16:07, Clara Kowalsky wrote:
>> If a tag and digest are specified for a container image in the SRC_URI,
>> the tag is ignored until now and the container image with the matching
>> digest is fetched.
>> With this change, the container image is fetched based on the specified
>> tag and it is checked whether the digest matches. If not, an error is
>> thrown.
>>
>> Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
>> ---
>>   meta/lib/container_fetcher.py | 17 +++++++++++++++++
>>   1 file changed, 17 insertions(+)
>>
>> diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
>> index 16467abb..08766742 100644
>> --- a/meta/lib/container_fetcher.py
>> +++ b/meta/lib/container_fetcher.py
>> @@ -11,6 +11,7 @@ from   bb.fetch2 import FetchMethod
>>   from   bb.fetch2 import logger
>>   from   bb.fetch2 import MissingChecksumEvent
>>   from   bb.fetch2 import NoChecksumError
>> +from   bb.fetch2 import ChecksumError
>>   from   bb.fetch2 import runfetchcmd
>>   
>>   class Container(FetchMethod):
>> @@ -47,6 +48,22 @@ class Container(FetchMethod):
>>       def download(self, ud, d):
>>           tarball = ud.localfile[:-len('.zst')]
>>           with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
>> +            # If both tag and digest are provided, verify they match
>> +            if ud.digest and not "tag" in ud.parm:
> 
> Hmm, I'm confused by my own suggestion right now: Did you test that
> again? Don't we rather need
> 
> if ud.digest and "tag" in ud.parm:
> 
> ?

Forgot to fully test this.. Yes, the v2 is not working. With "if 
ud.digest and not "tag" in ud.parm:", we only enter this case if no tag 
is specified in the SRC_URI, which is not what we want. The digest is 
then compared to the digest of "latest"...

Tested with "if ud.digest and "tag" in ud.parm:": This is correct, it 
means we have a tag and a digest explicitly specified in the SRC_URI and 
then we want to do the "skopeo inspect" to check for a mismatch.
Sending v3 with this change + Reviewed-By.

BR,
Clara

> 
>> +                inspect_output = runfetchcmd(f"skopeo inspect docker://{ud.container_name}:{ud.tag}", d, True)
>> +                actual_digest = json.loads(inspect_output)["Digest"]
>> +                if actual_digest != ud.digest:
>> +                    messages = []
>> +                    messages.append(f"Checksum mismatch for {ud.container_name}:{ud.tag}")
>> +                    messages.append("If this change is expected (e.g. you have upgraded " \
>> +                                "to a new version without updating the checksums) " \
>> +                                "then you can use these lines within the recipe:")
>> +                    messages.append(f'SRC_URI = "docker://{ud.container_name};digest={actual_digest};tag={ud.tag}"')
>> +                    messages.append("Otherwise you should retry the download and/or " \
>> +                                "check with upstream to determine if the container image has " \
>> +                                "become corrupted or otherwise unexpectedly modified.")
>> +                    raise ChecksumError("\n".join(messages), ud.url, actual_digest)
>> +
>>               # Take a two steps for downloading into a docker archive because
>>               # not all source may have the required Docker schema 2 manifest.
>>               runfetchcmd("skopeo copy --preserve-digests " + \
> 
> Jan
> 

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/953efeec-e0a3-41a9-a849-e5cc55bed601%40siemens.com.

Re: [PATCH v2 1/2] container_fetcher: Fix missing checksum warning

['Jan Kiszka' via isar-users]

On 26.06.25 16:07, Clara Kowalsky wrote:
> In case only a tag is specified for a container image in the SRC_URI and
> no digest, a warning should be issued with the recommendation to add the
> digest of the container image.
> So far, we were presenting in the warning the digest of the
> architecture-specific image that happened to be fetched first. However,
> we actually want to show the multi-arch manifest digest rather than the
> architecture-specific one.
> In addition, reading the manifest.json does not work at this point
> anyway, as skopeo has already packed it into a Docker archive.
> 
> Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
> ---
>  meta/lib/container_fetcher.py | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
> index 0d659154..16467abb 100644
> --- a/meta/lib/container_fetcher.py
> +++ b/meta/lib/container_fetcher.py
> @@ -6,6 +6,7 @@
>  import oe.path
>  import os
>  import tempfile
> +import json
>  from   bb.fetch2 import FetchMethod
>  from   bb.fetch2 import logger
>  from   bb.fetch2 import MissingChecksumEvent
> @@ -60,16 +61,17 @@ class Container(FetchMethod):
>          if ud.digest:
>              return
>  
> -        checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
> -        checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
> +        inspect_output = runfetchcmd(f"skopeo inspect docker://{ud.container_name}:{ud.tag}", d, True)
> +        digest = json.loads(inspect_output)["Digest"]
>  
> +        checksum_line = f'SRC_URI = "{ud.url};digest={digest}"'
>          strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
>  
>          # If strict checking enabled and neither sum defined, raise error
>          if strict == "1":
>              raise NoChecksumError(checksum_line)
>  
> -        checksum_event = {"sha256sum": checksum}
> +        checksum_event = {"sha256sum": digest}
>          bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
>  
>          if strict == "ignore":
> @@ -77,7 +79,7 @@ class Container(FetchMethod):
>  
>          # Log missing digest so user can more easily add it
>          logger.warning(
> -            f"Missing checksum for '{ud.localpath}', consider using this " \
> +            f"Missing checksum for '{ud.url}', consider using this " \
>              f"SRC_URI in the recipe:\n{checksum_line}")
>  
>      def unpack(self, ud, rootdir, d):

Reviewed-by: Jan Kiszka <jan.kiszka@siemens.com>

Thanks,
Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/8996932c-938e-4848-a4d6-5ccf0169a27e%40siemens.com.