[PATCH 0/2] Fix dpkg-raw usage of and

[PATCH 0/2] Fix dpkg-raw usage of and

[Jan Kiszka]

See patches for details.

Jan Kiszka (2):
  meta-isar: example-raw: Fix misuse of ${S}
  dpkg-raw: Correct path so that git fetching works

 meta-isar/recipes-app/example-raw/example-raw_0.3.bb |  4 ++--
 meta/classes/dpkg-raw.bbclass                        | 15 ++++++++++-----
 2 files changed, 12 insertions(+), 7 deletions(-)

-- 
2.16.4

[PATCH 1/2] meta-isar: example-raw: Fix misuse of ${S}

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We shall install into ${D}, not ${S}. This will break once dpkg-raw
makes both deviate.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-app/example-raw/example-raw_0.3.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta-isar/recipes-app/example-raw/example-raw_0.3.bb b/meta-isar/recipes-app/example-raw/example-raw_0.3.bb
index e4921709..21664642 100644
--- a/meta-isar/recipes-app/example-raw/example-raw_0.3.bb
+++ b/meta-isar/recipes-app/example-raw/example-raw_0.3.bb
@@ -35,10 +35,10 @@ do_install() {
 	bbnote "A HOME entry for everyone ... created after this is installed"
 	echo "hello skel" > ${WORKDIR}/${PN}-isar-skel.txt
 	install -v -d ${D}/etc/skel/
-	install -v -m 644 ${WORKDIR}/${PN}-isar-skel.txt ${S}/etc/skel/
+	install -v -m 644 ${WORKDIR}/${PN}-isar-skel.txt ${D}/etc/skel/
 
 	bbnote "A user-specific HOME entry"
 	echo "hello isar" > ${WORKDIR}/${PN}-isar.txt
 	install -v -d ${D}/var/lib/isar/
-	install -v -m 644 ${WORKDIR}/${PN}-isar.txt ${S}/var/lib/isar/
+	install -v -m 644 ${WORKDIR}/${PN}-isar.txt ${D}/var/lib/isar/
 }
-- 
2.16.4

[PATCH 2/2] dpkg-raw: Correct path so that git fetching works

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Setting D to S and cleaning the former caused problems when fetching the
to-be-installed files via git. Fix this by using a distinct D folder.
This requires setting --sourcedir for dh_install accordingly.

At this chance, simplify ${PN}.install creation and avoid resolving ${D}
in the error message.

Reported-by: Sven Schultschik <sven.schultschik@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 8c01a46a..9d4e8c19 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -5,22 +5,27 @@
 
 inherit dpkg
 
-D = "${S}"
+D = "${WORKDIR}/image"
 
 # Populate folder that will be picked up as package
 do_install() {
-	bbnote "Put your files for this package in ${D}"
+	bbnote "Put your files for this package in $""{D}"
 }
 
 do_install[cleandirs] = "${D}"
 addtask install after do_unpack before do_prepare_build
 
-do_prepare_build[cleandirs] += "${D}/debian"
+do_prepare_build[cleandirs] += "${S}/debian"
 do_prepare_build() {
 	cd ${D}
 	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name debian | \
-		sed 's:^./::' > ${WORKDIR}/${PN}.install
-	mv ${WORKDIR}/${PN}.install ${D}/debian/
+		sed 's:^./::' > ${S}/debian/${PN}.install
 
 	deb_debianize
+
+	cat <<EOF >> ${S}/debian/rules
+
+override_dh_install:
+	dh_install --sourcedir=${PP}/image
+EOF
 }
-- 
2.16.4

Re: [PATCH 0/2] Fix dpkg-raw usage of and

[Jan Kiszka]

On 03.04.20 17:02, [ext] Jan Kiszka wrote:
> See patches for details.
> 
> Jan Kiszka (2):
>    meta-isar: example-raw: Fix misuse of ${S}
>    dpkg-raw: Correct path so that git fetching works
> 
>   meta-isar/recipes-app/example-raw/example-raw_0.3.bb |  4 ++--
>   meta/classes/dpkg-raw.bbclass                        | 15 ++++++++++-----
>   2 files changed, 12 insertions(+), 7 deletions(-)
> 

FWIW, the subject line was "Fix dpkg-raw usage of ${S} and ${D}" before 
my submission script "resolved" them as vars.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 1/2] meta-isar: example-raw: Fix misuse of ${S}

[Henning Schild]

Am Fri, 3 Apr 2020 17:02:56 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> We shall install into ${D}, not ${S}. This will break once dpkg-raw
> makes both deviate.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta-isar/recipes-app/example-raw/example-raw_0.3.bb | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta-isar/recipes-app/example-raw/example-raw_0.3.bb
> b/meta-isar/recipes-app/example-raw/example-raw_0.3.bb index
> e4921709..21664642 100644 ---
> a/meta-isar/recipes-app/example-raw/example-raw_0.3.bb +++
> b/meta-isar/recipes-app/example-raw/example-raw_0.3.bb @@ -35,10
> +35,10 @@ do_install() { bbnote "A HOME entry for everyone ...
> created after this is installed" echo "hello skel" >
> ${WORKDIR}/${PN}-isar-skel.txt install -v -d ${D}/etc/skel/
> -	install -v -m 644 ${WORKDIR}/${PN}-isar-skel.txt
> ${S}/etc/skel/
> +	install -v -m 644 ${WORKDIR}/${PN}-isar-skel.txt
> ${D}/etc/skel/ 
>  	bbnote "A user-specific HOME entry"
>  	echo "hello isar" > ${WORKDIR}/${PN}-isar.txt
>  	install -v -d ${D}/var/lib/isar/
> -	install -v -m 644 ${WORKDIR}/${PN}-isar.txt
> ${S}/var/lib/isar/
> +	install -v -m 644 ${WORKDIR}/${PN}-isar.txt
> ${D}/var/lib/isar/ }

While this is correct it strongly suggests we will see similar problems
in downstream layers once the other patch gets applied.

We do not actually change the API, but i think an entry in the
changelog might still be a good idea.

Henning

Re: [PATCH 2/2] dpkg-raw: Correct path so that git fetching works

[Henning Schild]

Am Fri, 3 Apr 2020 17:02:57 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Setting D to S and cleaning the former caused problems when fetching
> the to-be-installed files via git. Fix this by using a distinct D
> folder. This requires setting --sourcedir for dh_install accordingly.
> 
> At this chance, simplify ${PN}.install creation and avoid resolving
> ${D} in the error message.
> 
> Reported-by: Sven Schultschik <sven.schultschik@siemens.com>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-raw.bbclass | 15 ++++++++++-----
>  1 file changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/meta/classes/dpkg-raw.bbclass
> b/meta/classes/dpkg-raw.bbclass index 8c01a46a..9d4e8c19 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -5,22 +5,27 @@
>  
>  inherit dpkg
>  
> -D = "${S}"
> +D = "${WORKDIR}/image"
>  
>  # Populate folder that will be picked up as package
>  do_install() {
> -	bbnote "Put your files for this package in ${D}"
> +	bbnote "Put your files for this package in $""{D}"
>  }
>  
>  do_install[cleandirs] = "${D}"
>  addtask install after do_unpack before do_prepare_build
>  
> -do_prepare_build[cleandirs] += "${D}/debian"
> +do_prepare_build[cleandirs] += "${S}/debian"
>  do_prepare_build() {
>  	cd ${D}

How about we create a recursive copy of S into D and leave the rest of
the code as it was?

>  	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name
> debian | \
> -		sed 's:^./::' > ${WORKDIR}/${PN}.install
> -	mv ${WORKDIR}/${PN}.install ${D}/debian/
> +		sed 's:^./::' > ${S}/debian/${PN}.install
>  
>  	deb_debianize
> +
> +	cat <<EOF >> ${S}/debian/rules
> +
> +override_dh_install:
> +	dh_install --sourcedir=${PP}/image
> +EOF

Because this looks confusing and will break if people already had their
own override ... for some weird reason.

And because of the hidden API "change" i commented on in the other
patch.

Henning

Re: [PATCH 2/2] dpkg-raw: Correct path so that git fetching works

[Henning Schild]

Am Tue, 14 Apr 2020 09:05:24 +0200
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Am Fri, 3 Apr 2020 17:02:57 +0200
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
> > From: Jan Kiszka <jan.kiszka@siemens.com>
> > 
> > Setting D to S and cleaning the former caused problems when fetching
> > the to-be-installed files via git. Fix this by using a distinct D
> > folder. This requires setting --sourcedir for dh_install
> > accordingly.
> > 
> > At this chance, simplify ${PN}.install creation and avoid resolving
> > ${D} in the error message.
> > 
> > Reported-by: Sven Schultschik <sven.schultschik@siemens.com>
> > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > ---
> >  meta/classes/dpkg-raw.bbclass | 15 ++++++++++-----
> >  1 file changed, 10 insertions(+), 5 deletions(-)
> > 
> > diff --git a/meta/classes/dpkg-raw.bbclass
> > b/meta/classes/dpkg-raw.bbclass index 8c01a46a..9d4e8c19 100644
> > --- a/meta/classes/dpkg-raw.bbclass
> > +++ b/meta/classes/dpkg-raw.bbclass
> > @@ -5,22 +5,27 @@
> >  
> >  inherit dpkg
> >  
> > -D = "${S}"
> > +D = "${WORKDIR}/image"
> >  
> >  # Populate folder that will be picked up as package
> >  do_install() {
> > -	bbnote "Put your files for this package in ${D}"
> > +	bbnote "Put your files for this package in $""{D}"
> >  }
> >  
> >  do_install[cleandirs] = "${D}"
> >  addtask install after do_unpack before do_prepare_build
> >  
> > -do_prepare_build[cleandirs] += "${D}/debian"
> > +do_prepare_build[cleandirs] += "${S}/debian"
> >  do_prepare_build() {
> >  	cd ${D}  
> 
> How about we create a recursive copy of S into D and leave the rest of
> the code as it was?

In fact the debianization and creation of that install file are
something like a patch step. We can either try to roll it back by
having a reverse implementation ... maybe actually using quilt.
Or we need to invalidate its stamp together with unpack.

Henning

> >  	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name
> > debian | \
> > -		sed 's:^./::' > ${WORKDIR}/${PN}.install
> > -	mv ${WORKDIR}/${PN}.install ${D}/debian/
> > +		sed 's:^./::' > ${S}/debian/${PN}.install
> >  
> >  	deb_debianize
> > +
> > +	cat <<EOF >> ${S}/debian/rules
> > +
> > +override_dh_install:
> > +	dh_install --sourcedir=${PP}/image
> > +EOF  
> 
> Because this looks confusing and will break if people already had
> their own override ... for some weird reason.
> 
> And because of the hidden API "change" i commented on in the other
> patch.
> 
> Henning
> 

Re: [PATCH 2/2] dpkg-raw: Correct path so that git fetching works

[Jan Kiszka]

On 14.04.20 09:05, Henning Schild wrote:
> Am Fri, 3 Apr 2020 17:02:57 +0200
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Setting D to S and cleaning the former caused problems when fetching
>> the to-be-installed files via git. Fix this by using a distinct D
>> folder. This requires setting --sourcedir for dh_install accordingly.
>>
>> At this chance, simplify ${PN}.install creation and avoid resolving
>> ${D} in the error message.
>>
>> Reported-by: Sven Schultschik <sven.schultschik@siemens.com>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/dpkg-raw.bbclass | 15 ++++++++++-----
>>   1 file changed, 10 insertions(+), 5 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-raw.bbclass
>> b/meta/classes/dpkg-raw.bbclass index 8c01a46a..9d4e8c19 100644
>> --- a/meta/classes/dpkg-raw.bbclass
>> +++ b/meta/classes/dpkg-raw.bbclass
>> @@ -5,22 +5,27 @@
>>   
>>   inherit dpkg
>>   
>> -D = "${S}"
>> +D = "${WORKDIR}/image"
>>   
>>   # Populate folder that will be picked up as package
>>   do_install() {
>> -	bbnote "Put your files for this package in ${D}"
>> +	bbnote "Put your files for this package in $""{D}"
>>   }
>>   
>>   do_install[cleandirs] = "${D}"
>>   addtask install after do_unpack before do_prepare_build
>>   
>> -do_prepare_build[cleandirs] += "${D}/debian"
>> +do_prepare_build[cleandirs] += "${S}/debian"
>>   do_prepare_build() {
>>   	cd ${D}
> 
> How about we create a recursive copy of S into D and leave the rest of
> the code as it was?

This may cause artifacts being distributed that shall not. ${S} may 
contain a complete git repo while ${D} shall only contain some selected 
files of it.

> 
>>   	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name
>> debian | \
>> -		sed 's:^./::' > ${WORKDIR}/${PN}.install
>> -	mv ${WORKDIR}/${PN}.install ${D}/debian/
>> +		sed 's:^./::' > ${S}/debian/${PN}.install
>>   
>>   	deb_debianize
>> +
>> +	cat <<EOF >> ${S}/debian/rules
>> +
>> +override_dh_install:
>> +	dh_install --sourcedir=${PP}/image
>> +EOF
> 
> Because this looks confusing and will break if people already had their
> own override ... for some weird reason.
> 
> And because of the hidden API "change" i commented on in the other
> patch.

I can write a small changelog entry, but I don't see any point in 
addressing purely hypothetical user issues by copying stuff around - 
which may even not be as simple as you think.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH v2 2/2] dpkg-raw: Correct path so that git fetching works

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Setting D to S and cleaning the former caused problems when fetching the
to-be-installed files via git. Fix this by using a distinct D folder.
This requires setting --sourcedir for dh_install accordingly.

At this chance, simplify ${PN}.install creation and avoid resolving ${D}
in the error message.

Reported-by: Sven Schultschik <sven.schultschik@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v2:
 - add RECIPE-API-CHANGELOG entry

 RECIPE-API-CHANGELOG.md       |  5 +++++
 meta/classes/dpkg-raw.bbclass | 15 ++++++++++-----
 2 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index fc416a4f..5717daad 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -236,3 +236,8 @@ downloads from previous builds for the cache.
 
 DTB_FILES now allows to specify multiple DTBs that should be deployed for
 consumption by imaging classes.
+
+### Separation of ${S} and ${D} in dpkg-raw
+
+${S} can now be used for checking out sources without being linked implicitly
+with ${D} which needs to be filled explicitly in do_install as before.
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 8c01a46a..9d4e8c19 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -5,22 +5,27 @@
 
 inherit dpkg
 
-D = "${S}"
+D = "${WORKDIR}/image"
 
 # Populate folder that will be picked up as package
 do_install() {
-	bbnote "Put your files for this package in ${D}"
+	bbnote "Put your files for this package in $""{D}"
 }
 
 do_install[cleandirs] = "${D}"
 addtask install after do_unpack before do_prepare_build
 
-do_prepare_build[cleandirs] += "${D}/debian"
+do_prepare_build[cleandirs] += "${S}/debian"
 do_prepare_build() {
 	cd ${D}
 	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name debian | \
-		sed 's:^./::' > ${WORKDIR}/${PN}.install
-	mv ${WORKDIR}/${PN}.install ${D}/debian/
+		sed 's:^./::' > ${S}/debian/${PN}.install
 
 	deb_debianize
+
+	cat <<EOF >> ${S}/debian/rules
+
+override_dh_install:
+	dh_install --sourcedir=${PP}/image
+EOF
 }
-- 
2.16.4

Re: [PATCH 2/2] dpkg-raw: Correct path so that git fetching works

[Jan Kiszka]

On 14.04.20 09:11, Henning Schild wrote:
> Am Tue, 14 Apr 2020 09:05:24 +0200
> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> 
>> Am Fri, 3 Apr 2020 17:02:57 +0200
>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> Setting D to S and cleaning the former caused problems when fetching
>>> the to-be-installed files via git. Fix this by using a distinct D
>>> folder. This requires setting --sourcedir for dh_install
>>> accordingly.
>>>
>>> At this chance, simplify ${PN}.install creation and avoid resolving
>>> ${D} in the error message.
>>>
>>> Reported-by: Sven Schultschik <sven.schultschik@siemens.com>
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>   meta/classes/dpkg-raw.bbclass | 15 ++++++++++-----
>>>   1 file changed, 10 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/meta/classes/dpkg-raw.bbclass
>>> b/meta/classes/dpkg-raw.bbclass index 8c01a46a..9d4e8c19 100644
>>> --- a/meta/classes/dpkg-raw.bbclass
>>> +++ b/meta/classes/dpkg-raw.bbclass
>>> @@ -5,22 +5,27 @@
>>>   
>>>   inherit dpkg
>>>   
>>> -D = "${S}"
>>> +D = "${WORKDIR}/image"
>>>   
>>>   # Populate folder that will be picked up as package
>>>   do_install() {
>>> -	bbnote "Put your files for this package in ${D}"
>>> +	bbnote "Put your files for this package in $""{D}"
>>>   }
>>>   
>>>   do_install[cleandirs] = "${D}"
>>>   addtask install after do_unpack before do_prepare_build
>>>   
>>> -do_prepare_build[cleandirs] += "${D}/debian"
>>> +do_prepare_build[cleandirs] += "${S}/debian"
>>>   do_prepare_build() {
>>>   	cd ${D}
>>
>> How about we create a recursive copy of S into D and leave the rest of
>> the code as it was?
> 
> In fact the debianization and creation of that install file are
> something like a patch step. We can either try to roll it back by
> having a reverse implementation ... maybe actually using quilt.
> Or we need to invalidate its stamp together with unpack.

As explained in my other email: It's not only about adding debian/, this 
is also about picking from ${S} into ${D}. You are still on the old path 
with your proposal.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 2/2] dpkg-raw: Correct path so that git fetching works

[Baurzhan Ismagulov]

On Tue, Apr 14, 2020 at 09:34:05AM +0200, Jan Kiszka wrote:
> Setting D to S and cleaning the former caused problems when fetching the
> to-be-installed files via git. Fix this by using a distinct D folder.
> This requires setting --sourcedir for dh_install accordingly.
> 
> At this chance, simplify ${PN}.install creation and avoid resolving ${D}
> in the error message.

Looks good to me.

With kind regards,
Baurzhan.

Re: [PATCH v2 2/2] dpkg-raw: Correct path so that git fetching works

[Baurzhan Ismagulov]

On Tue, Apr 14, 2020 at 09:34:05AM +0200, Jan Kiszka wrote:
> Setting D to S and cleaning the former caused problems when fetching the
> to-be-installed files via git. Fix this by using a distinct D folder.
> This requires setting --sourcedir for dh_install accordingly.
> 
> At this chance, simplify ${PN}.install creation and avoid resolving ${D}
> in the error message.

Applied to next, thanks.

With kind regards,
Baurzhan.