[PATCH] Allow disabling source packages cleanup

[PATCH] Allow disabling source packages cleanup

[Anton Mikanovich]

To allow building source packages containing binary or CVS files
introduce DPKG_SOURCE_EXTRA_ARGS variable can be set per recipe.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 RECIPE-API-CHANGELOG.md          | 15 +++++++++++++++
 meta/classes/dpkg-source.bbclass |  4 +++-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7d463473..b1699be6 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -539,3 +539,18 @@ some cases dpkg_runbuild:prepend should be replaced by do_dpkg_source:prepend.
 We need local copy of isar-apt to have build dependencies reachable. Now is
 prepared in separate task: do_local_isarapt.
 This task depends of do_deploy_deb of all build dependency recipes.
+
+### Skipping source package cleanup
+
+By default Isar filter out control files and directories of the most common
+revision control systems, backup and swap files and Libtool build output
+directories from the source package.
+Now this can be overriden by setting DPKG_SOURCE_EXTRA_ARGS value in recipe.
+
+Default value is '-I' which sets filter to:
+
+*.a -I*.la -I*.o -I*.so -I.*.sw? -I*/*~ -I,,* -I.[#~]* -I.arch-ids
+-I.arch-inventory -I.be -I.bzr -I.bzr.backup -I.bzr.tags -I.bzrignore
+-I.cvsignore -I.deps -I.git -I.gitattributes -I.gitignore -I.gitmodules
+-I.gitreview -I.hg -I.hgignore -I.hgsigs -I.hgtags -I.mailmap -I.mtn-ignore
+-I.shelf -I.svn -ICVS -IDEADJOE -IRCS -I_MTN -I_darcs -I{arch}
diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 1f4f8164..7e3868f6 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -5,6 +5,8 @@
 
 inherit dpkg-base
 
+DPKG_SOURCE_EXTRA_ARGS ?= "-I"
+
 do_dpkg_source() {
     # Create a .dsc file from source directory to use it with sbuild
     DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
@@ -12,7 +14,7 @@ do_dpkg_source() {
     if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
         export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
     fi
-    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
+    sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
 }
 addtask dpkg_source after do_prepare_build before do_dpkg_build
 
-- 
2.34.1

Re: [PATCH] Allow disabling source packages cleanup

[Jan Kiszka]

On 12.10.23 17:01, Anton Mikanovich wrote:
> To allow building source packages containing binary or CVS files
> introduce DPKG_SOURCE_EXTRA_ARGS variable can be set per recipe.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  RECIPE-API-CHANGELOG.md          | 15 +++++++++++++++
>  meta/classes/dpkg-source.bbclass |  4 +++-
>  2 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 7d463473..b1699be6 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -539,3 +539,18 @@ some cases dpkg_runbuild:prepend should be replaced by do_dpkg_source:prepend.
>  We need local copy of isar-apt to have build dependencies reachable. Now is
>  prepared in separate task: do_local_isarapt.
>  This task depends of do_deploy_deb of all build dependency recipes.
> +
> +### Skipping source package cleanup
> +
> +By default Isar filter out control files and directories of the most common
> +revision control systems, backup and swap files and Libtool build output
> +directories from the source package.
> +Now this can be overriden by setting DPKG_SOURCE_EXTRA_ARGS value in recipe.
> +
> +Default value is '-I' which sets filter to:
> +
> +*.a -I*.la -I*.o -I*.so -I.*.sw? -I*/*~ -I,,* -I.[#~]* -I.arch-ids
> +-I.arch-inventory -I.be -I.bzr -I.bzr.backup -I.bzr.tags -I.bzrignore
> +-I.cvsignore -I.deps -I.git -I.gitattributes -I.gitignore -I.gitmodules
> +-I.gitreview -I.hg -I.hgignore -I.hgsigs -I.hgtags -I.mailmap -I.mtn-ignore
> +-I.shelf -I.svn -ICVS -IDEADJOE -IRCS -I_MTN -I_darcs -I{arch}

Where is dpkg-source documenting this list? Or does it come from
somewhere else?

I'm also asking because could be useful to reduce it by certain patterns
(or maybe even files) so that you can still benefit from filtering, say,
version control stuff while keeping other needed files (in our case some
*.a files).

> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
> index 1f4f8164..7e3868f6 100644
> --- a/meta/classes/dpkg-source.bbclass
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -5,6 +5,8 @@
>  
>  inherit dpkg-base
>  
> +DPKG_SOURCE_EXTRA_ARGS ?= "-I"
> +
>  do_dpkg_source() {
>      # Create a .dsc file from source directory to use it with sbuild
>      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> @@ -12,7 +14,7 @@ do_dpkg_source() {
>      if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>          export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>      fi
> -    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
> +    sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
>  }
>  addtask dpkg_source after do_prepare_build before do_dpkg_build
>  

This would mitigate the current issue as well, but I'd be careful to not
hurry with introducing a half-baked interface until we have thought
through some use cases.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH] Allow disabling source packages cleanup

[Uladzimir Bely]

On Thu, 2023-10-12 at 17:21 +0200, 'Jan Kiszka' via isar-users wrote:
> On 12.10.23 17:01, Anton Mikanovich wrote:
> > To allow building source packages containing binary or CVS files
> > introduce DPKG_SOURCE_EXTRA_ARGS variable can be set per recipe.
> > 
> > Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > ---
> >  RECIPE-API-CHANGELOG.md          | 15 +++++++++++++++
> >  meta/classes/dpkg-source.bbclass |  4 +++-
> >  2 files changed, 18 insertions(+), 1 deletion(-)
> > 
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 7d463473..b1699be6 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -539,3 +539,18 @@ some cases dpkg_runbuild:prepend should be
> > replaced by do_dpkg_source:prepend.
> >  We need local copy of isar-apt to have build dependencies
> > reachable. Now is
> >  prepared in separate task: do_local_isarapt.
> >  This task depends of do_deploy_deb of all build dependency
> > recipes.
> > +
> > +### Skipping source package cleanup
> > +
> > +By default Isar filter out control files and directories of the
> > most common
> > +revision control systems, backup and swap files and Libtool build
> > output
> > +directories from the source package.
> > +Now this can be overriden by setting DPKG_SOURCE_EXTRA_ARGS value
> > in recipe.
> > +
> > +Default value is '-I' which sets filter to:
> > +
> > +*.a -I*.la -I*.o -I*.so -I.*.sw? -I*/*~ -I,,* -I.[#~]* -I.arch-ids
> > +-I.arch-inventory -I.be -I.bzr -I.bzr.backup -I.bzr.tags -
> > I.bzrignore
> > +-I.cvsignore -I.deps -I.git -I.gitattributes -I.gitignore -
> > I.gitmodules
> > +-I.gitreview -I.hg -I.hgignore -I.hgsigs -I.hgtags -I.mailmap -
> > I.mtn-ignore
> > +-I.shelf -I.svn -ICVS -IDEADJOE -IRCS -I_MTN -I_darcs -I{arch}
> 
> Where is dpkg-source documenting this list? Or does it come from
> somewhere else?
> 

This is from "dpkg-source --help".

> I'm also asking because could be useful to reduce it by certain
> patterns
> (or maybe even files) so that you can still benefit from filtering,
> say,
> version control stuff while keeping other needed files (in our case
> some
> *.a files).
> 
> > diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-
> > source.bbclass
> > index 1f4f8164..7e3868f6 100644
> > --- a/meta/classes/dpkg-source.bbclass
> > +++ b/meta/classes/dpkg-source.bbclass
> > @@ -5,6 +5,8 @@
> >  
> >  inherit dpkg-base
> >  
> > +DPKG_SOURCE_EXTRA_ARGS ?= "-I"
> > +
> >  do_dpkg_source() {
> >      # Create a .dsc file from source directory to use it with
> > sbuild
> >      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --
> > file ${WORKDIR}/${PPS}/debian/changelog)
> > @@ -12,7 +14,7 @@ do_dpkg_source() {
> >      if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
> >          export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
> >      fi
> > -    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
> > +    sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b
> > ${PPS}"
> >  }
> >  addtask dpkg_source after do_prepare_build before do_dpkg_build
> >  
> 
> This would mitigate the current issue as well, but I'd be careful to
> not
> hurry with introducing a half-baked interface until we have thought
> through some use cases.
> 
> Jan
> 
> -- 
> Siemens AG, Technology
> Linux Expert Center
> 

Re: [PATCH] Allow disabling source packages cleanup

[Jan Kiszka]

On 12.10.23 17:57, Uladzimir Bely wrote:
> On Thu, 2023-10-12 at 17:21 +0200, 'Jan Kiszka' via isar-users wrote:
>> On 12.10.23 17:01, Anton Mikanovich wrote:
>>> To allow building source packages containing binary or CVS files
>>> introduce DPKG_SOURCE_EXTRA_ARGS variable can be set per recipe.
>>>
>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>> ---
>>>  RECIPE-API-CHANGELOG.md          | 15 +++++++++++++++
>>>  meta/classes/dpkg-source.bbclass |  4 +++-
>>>  2 files changed, 18 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
>>> index 7d463473..b1699be6 100644
>>> --- a/RECIPE-API-CHANGELOG.md
>>> +++ b/RECIPE-API-CHANGELOG.md
>>> @@ -539,3 +539,18 @@ some cases dpkg_runbuild:prepend should be
>>> replaced by do_dpkg_source:prepend.
>>>  We need local copy of isar-apt to have build dependencies
>>> reachable. Now is
>>>  prepared in separate task: do_local_isarapt.
>>>  This task depends of do_deploy_deb of all build dependency
>>> recipes.
>>> +
>>> +### Skipping source package cleanup
>>> +
>>> +By default Isar filter out control files and directories of the
>>> most common
>>> +revision control systems, backup and swap files and Libtool build
>>> output
>>> +directories from the source package.
>>> +Now this can be overriden by setting DPKG_SOURCE_EXTRA_ARGS value
>>> in recipe.
>>> +
>>> +Default value is '-I' which sets filter to:
>>> +
>>> +*.a -I*.la -I*.o -I*.so -I.*.sw? -I*/*~ -I,,* -I.[#~]* -I.arch-ids
>>> +-I.arch-inventory -I.be -I.bzr -I.bzr.backup -I.bzr.tags -
>>> I.bzrignore
>>> +-I.cvsignore -I.deps -I.git -I.gitattributes -I.gitignore -
>>> I.gitmodules
>>> +-I.gitreview -I.hg -I.hgignore -I.hgsigs -I.hgtags -I.mailmap -
>>> I.mtn-ignore
>>> +-I.shelf -I.svn -ICVS -IDEADJOE -IRCS -I_MTN -I_darcs -I{arch}
>>
>> Where is dpkg-source documenting this list? Or does it come from
>> somewhere else?
>>
> 
> This is from "dpkg-source --help".
> 
>> I'm also asking because could be useful to reduce it by certain
>> patterns
>> (or maybe even files) so that you can still benefit from filtering,
>> say,
>> version control stuff while keeping other needed files (in our case
>> some
>> *.a files).
>>
>>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-
>>> source.bbclass
>>> index 1f4f8164..7e3868f6 100644
>>> --- a/meta/classes/dpkg-source.bbclass
>>> +++ b/meta/classes/dpkg-source.bbclass
>>> @@ -5,6 +5,8 @@
>>>  
>>>  inherit dpkg-base
>>>  
>>> +DPKG_SOURCE_EXTRA_ARGS ?= "-I"
>>> +
>>>  do_dpkg_source() {
>>>      # Create a .dsc file from source directory to use it with
>>> sbuild
>>>      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --
>>> file ${WORKDIR}/${PPS}/debian/changelog)
>>> @@ -12,7 +14,7 @@ do_dpkg_source() {
>>>      if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>>>          export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>>>      fi
>>> -    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>>> +    sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b
>>> ${PPS}"
>>>  }
>>>  addtask dpkg_source after do_prepare_build before do_dpkg_build
>>>  
>>
>> This would mitigate the current issue as well, but I'd be careful to
>> not
>> hurry with introducing a half-baked interface until we have thought
>> through some use cases.
>>

I've checked if there could be other, better, options forcing us to
change this again, but it seems this is the only reasonable way forward.
So I'm adding my

Reviewed-by: Jan Kiszka <jan.kiszka@siemens.com>

and I'm looking forward to picking this up from the tree to deviate with
DPKG_SOURCE_EXTRA_ARGS in our recipe. Let's hope we were widely alone
with this issue...

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH] Allow disabling source packages cleanup

[Uladzimir Bely]

On Thu, 2023-10-12 at 18:01 +0300, Anton Mikanovich wrote:
> To allow building source packages containing binary or CVS files
> introduce DPKG_SOURCE_EXTRA_ARGS variable can be set per recipe.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  RECIPE-API-CHANGELOG.md          | 15 +++++++++++++++
>  meta/classes/dpkg-source.bbclass |  4 +++-
>  2 files changed, 18 insertions(+), 1 deletion(-)
> 

Applied to next.