RE: [PATCH v5 18/21] isar-sstate: adopt sstate maintenance script

["Schmidt, Adriaan" <adriaan.schmidt@siemens.com>]

Moessbauer, Felix (T CED INW-CN), Samstag, 3. Dezember 2022 05:58:
> On Fri, 2022-12-02 at 17:36 +0300, Anton Mikanovich wrote:
> > Fix sstate maintenance helper script to work with updated Bitbake
> > siginfo data format.
> >
> > Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > ---
> >  scripts/isar-sstate | 70 ++++++++++++++++++++++---------------------
> > --
> >  1 file changed, 35 insertions(+), 35 deletions(-)
> >
> > diff --git a/scripts/isar-sstate b/scripts/isar-sstate
> > index 8908b09f..53d0541f 100755
> > --- a/scripts/isar-sstate
> > +++ b/scripts/isar-sstate
> > @@ -126,7 +126,7 @@ import shutil
> >  import sys
> >  from tempfile import NamedTemporaryFile
> >  import time
> > -import pickle
> > +import json
> >
> >  sys.path.insert(0,
> > os.path.join(os.path.dirname(os.path.realpath(__file__)), '..',
> > 'bitbake', 'lib'))
> >  from bb.siggen import compare_sigfiles
> > @@ -819,41 +819,41 @@ def sstate_lint(target, verbose, sources_dir,
> > build_dir, exit_code, pedantic, **
> >              continue
> >
> >          sig_file = target.download(sig.path)
> > -        with open(sig_file, 'rb') as f:
> > -            try:
> > -                sigdata_raw = pickle.Unpickler(f)
> > -                sigdata = sigdata_raw.load()
> > -            except:
> > -                # invalid file or format... never mind
> > -                continue
> > +        try:
> > +            with bb.compress.zstd.open(sig_file, "rt",
> > encoding="utf-8", num_threads=1) as f:
> > +                sigdata = json.load(f,
> > object_hook=bb.siggen.SetDecoder)
> > +            bb.siggen.handle_renames(sigdata)
> > +        except:
> > +            # invalid file or format... never mind
> > +            continue
> 
> Hi, I guess we need a similar logic for the analyze subcommand? There,
> I see a lot of these messages (even before the first "checking local
> item" output):
> 
> zstd: /*stdin*\: unsupported format

In the analyze subcommand we don't open the files ourselves, but use bitbake's
compare_sigfiles(), which runs zstd as a new process. We do catch the exceptions,
but not before zstd gets to print that message to stderr (I guess even with this
logic we would still see those).
I don't think it's worth fixing this. It only happens if you try to run the
new script on pre-bitbake2.0 sstate artifacts.

Adriaan

> Felix
> 
> >
> > -            pn_issues = []
> > -            for name, val in sigdata['varvals'].items():
> > -                if not name[0].isupper():
> > -                    continue
> > -                if sigdata['basewhitelist'] and name in
> > sigdata['basewhitelist'] or \
> > -                   sigdata['taskwhitelist'] and name in
> > sigdata['taskwhitelist'] or \
> > -                   name in ADDITIONAL_IGNORED_VARNAMES:
> > -                    continue
> > -                if not val:
> > -                    continue
> > -                # remove leading whitespaces possibly added by
> > appending
> > -                val = val.lstrip()
> > -                if not val[0] == '/':
> > -                    continue
> > -                if val.startswith(build_dir):
> > -                    pn_issues.append(f'\033[0;31m-> path in build-
> > dir:   {name} = "{val}"\033[0m')
> > -                    hits_builddir += 1
> > -                elif val.startswith(sources_dir):
> > -                    pn_issues.append(f'\033[0;31m-> path in sources-
> > dir: {name} = "{val}"\033[0m')
> > -                    hits_srcdir += 1
> > -                else:
> > -                    hits_other += 1
> > -                    if verbose:
> > -                        pn_issues.append(f'\033[0;34m-> other
> > absolute path: {name} = "{val}"\033[0m')
> > -            if len(pn_issues) > 0:
> > -                print(f"\033[1;33m==== issues found in
> > {sig.arch}:{sig.pn}:{sig.task} ({sig.hash[:8]}) ====\033[0m")
> > -                print('\n'.join(pn_issues))
> > +        pn_issues = []
> > +        for name, val in sigdata['varvals'].items():
> > +            if not name[0].isupper():
> > +                continue
> > +            if sigdata['basehash_ignore_vars'] and name in
> > sigdata['basehash_ignore_vars'] or \
> > +               sigdata['taskhash_ignore_tasks'] and name in
> > sigdata['taskhash_ignore_tasks'] or \
> > +               name in ADDITIONAL_IGNORED_VARNAMES:
> > +                continue
> > +            if not val:
> > +                continue
> > +            # remove leading whitespaces possibly added by appending
> > +            val = val.lstrip()
> > +            if not val[0] == '/':
> > +                continue
> > +            if val.startswith(build_dir):
> > +                pn_issues.append(f'\033[0;31m-> path in build-dir:
> > {name} = "{val}"\033[0m')
> > +                hits_builddir += 1
> > +            elif val.startswith(sources_dir):
> > +                pn_issues.append(f'\033[0;31m-> path in sources-dir:
> > {name} = "{val}"\033[0m')
> > +                hits_srcdir += 1
> > +            else:
> > +                hits_other += 1
> > +                if verbose:
> > +                    pn_issues.append(f'\033[0;34m-> other absolute
> > path: {name} = "{val}"\033[0m')
> > +        if len(pn_issues) > 0:
> > +            print(f"\033[1;33m==== issues found in
> > {sig.arch}:{sig.pn}:{sig.task} ({sig.hash[:8]}) ====\033[0m")
> > +            print('\n'.join(pn_issues))
> >          target.release(sig_file)
> >
> >      sum_hits = hits_srcdir + hits_builddir
> > --
> > 2.17.1
> >
>