[isar] reproducible build failures

[isar] reproducible build failures

[Venkata.Pyla]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

Hi,

I am using isar system in isar-cip-core project [1] where I found some reproducible failures, which may be good to fix in the isar system.
I am not good in modifying the isar system, so could you please guide me to fix these problems?

Here are the steps to check the reproducible failures in isar-cip-core project:
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/12
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/13

I also verified the reproducibility in the isar system and found similar failures that are copied below:
==============================================
tmp/gpghomefHv8eRhk43/
tmp/gpghomefHv8eRhk43/private-keys-v1.d/
usr/share/doc/hello/changelog.Debian.gz
var/cache/debconf/config.dat
var/cache/debconf/config.dat-old
var/cache/ldconfig/aux-cache
var/lib/dpkg/info/enable-fsck.md5sums
var/lib/dpkg/info/example-raw.md5sums
var/lib/dpkg/info/hello.md5sums
var/lib/dpkg/info/isar-disable-apt-cache.md5sums
var/lib/dpkg/info/isar-exclude-docs.md5sums
var/lib/dpkg/info/sshd-regen-keys.md5sums
var/lib/initramfs-tools/4.19.0-17-amd64
var/lib/systemd/catalog/database
var/log/alternatives.log
var/log/bootstrap.log
var/log/dpkg.log
var/log/apt/history.log
var/log/apt/term.log
==============================================

Steps to check reproducible failures in isar
====================================
$ . isar-init-build-env ../build1 && bitbake mc:qemuamd64-buster-tgz:isar-image-base
$  . isar-init-build-env ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base
$ mkdir -p rootfs1 rootfs2
$ tar -xzvf ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz -C ./rootfs1/
$ tar -xzvf ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz -C ./rootfs2/
$ rsync -nrclv ./rootfs1/ ./rootfs2/ > difference.txt
====================================

>From the reproducible failures I found there are three different areas to fix these problem

1.       Changelog file generation, which is embedding the build time date value at here (https://github.com/ilbers/isar/blob/master/meta/classes/debianize.bbclass#L34 )

2.       Log files generated by different application, which are adding build time values, I think we can remove these files if it is not required after build. ( I tried at here https://github.com/ilbers/isar/blob/master/meta/classes/image.bbclass#L183 but it did not work)

3.       Cache and temporary files, I think we can delete these files also.

Please guide me to fix these issues.

Thanks,
Venkata.

[-- Attachment #2: Type: text/html, Size: 9749 bytes --]

Re: [isar] reproducible build failures

[Jan Kiszka]

Hi Venkata,

please avoid HTML emails on mailing lists.

On 03.09.21 17:19, Venkata.Pyla@toshiba-tsip.com wrote:
> Hi,
> 
> I am using isar system in isar-cip-core project [1] where I found some
> reproducible failures, which may be good to fix in the isar system.
> I am not good in modifying the isar system, so could you please guide me
> to fix these problems?
> 
> Here are the steps to check the reproducible failures in isar-cip-core
> project:
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/12
> <https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/12>
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/13
> <https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/13>
> 
> I also verified the reproducibility in the isar system and found similar
> failures that are copied below:
> ==============================================
> tmp/gpghomefHv8eRhk43/
> tmp/gpghomefHv8eRhk43/private-keys-v1.d/
> usr/share/doc/hello/changelog.Debian.gz
> var/cache/debconf/config.dat
> var/cache/debconf/config.dat-old
> var/cache/ldconfig/aux-cache
> var/lib/dpkg/info/enable-fsck.md5sums
> var/lib/dpkg/info/example-raw.md5sums
> var/lib/dpkg/info/hello.md5sums
> var/lib/dpkg/info/isar-disable-apt-cache.md5sums
> var/lib/dpkg/info/isar-exclude-docs.md5sums
> var/lib/dpkg/info/sshd-regen-keys.md5sums
> var/lib/initramfs-tools/4.19.0-17-amd64
> var/lib/systemd/catalog/database
> var/log/alternatives.log
> var/log/bootstrap.log
> var/log/dpkg.log
> var/log/apt/history.log
> var/log/apt/term.log
> ==============================================
> 
> Steps to check reproducible failures in isar
> ====================================
> $ . isar-init-build-env ../build1 && bitbake mc:qemuamd64-buster-tgz:isar-image-base
> $  . isar-init-build-env ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base
> $ mkdir -p rootfs1 rootfs2
> $ tar -xzvf ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz -C ./rootfs1/
> $ tar -xzvf ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz -C ./rootfs2/
> $ rsync -nrclv ./rootfs1/ ./rootfs2/ > difference.txt
> ====================================  
>  
> From the reproducible failures I found there are three different areas
> to fix these problem
> 1.       Changelog file generation, which is embedding the build time
> date value at here
> (https://github.com/ilbers/isar/blob/master/meta/classes/debianize.bbclass#L34
> <https://github.com/ilbers/isar/blob/master/meta/classes/debianize.bbclass#L34>
> )
> 2.       Log files generated by different application, which are adding
> build time values, I think we can remove these files if it is not
> required after build. ( I tried at here
> https://github.com/ilbers/isar/blob/master/meta/classes/image.bbclass#L183
> <https://github.com/ilbers/isar/blob/master/meta/classes/image.bbclass#L183>
> but it did not work)
> 3.       Cache and temporary files, I think we can delete these files also.
> 
> Please guide me to fix these issues.
> 

Did you also cross-check what
https://wiki.debian.org/ReproducibleInstalls suggests?

Then maybe start with the [isar-]bootstrap. Hook into
https://github.com/ilbers/isar/blob/03124cca669f50b682336a0bdf4ede5a4238e144/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc#L232
to perform cleanups.

Using SOURCE_DATE_EPOCH as control over timestamps might be an interface
worth to explore.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [isar] reproducible build failures

[Henning Schild]

Hi there,

Am Fri, 3 Sep 2021 15:19:21 +0000
schrieb <Venkata.Pyla@toshiba-tsip.com>:

> Hi,
> 
> I am using isar system in isar-cip-core project [1] where I found
> some reproducible failures, which may be good to fix in the isar
> system. I am not good in modifying the isar system, so could you
> please guide me to fix these problems?

Well ... for isar and maybe to some degree also to debian, a truly
reproducible build would be a new topic that so far has been ignored.

> Here are the steps to check the reproducible failures in
> isar-cip-core project:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F12&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ofcsE0TXKbwL%2FPUTD2hAEQWvOLXNRFrIvunSOblUAho%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F13&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=M7pZHme2TzX7ZhNbaZW%2BKn26K65ZnKFW4BvnQJldLnw%3D&amp;reserved=0
> 
> I also verified the reproducibility in the isar system and found
> similar failures that are copied below:
> ============================================== tmp/gpghomefHv8eRhk43/
> tmp/gpghomefHv8eRhk43/private-keys-v1.d/
> usr/share/doc/hello/changelog.Debian.gz
> var/cache/debconf/config.dat
> var/cache/debconf/config.dat-old
> var/cache/ldconfig/aux-cache
> var/lib/dpkg/info/enable-fsck.md5sums
> var/lib/dpkg/info/example-raw.md5sums
> var/lib/dpkg/info/hello.md5sums
> var/lib/dpkg/info/isar-disable-apt-cache.md5sums
> var/lib/dpkg/info/isar-exclude-docs.md5sums
> var/lib/dpkg/info/sshd-regen-keys.md5sums
> var/lib/initramfs-tools/4.19.0-17-amd64
> var/lib/systemd/catalog/database
> var/log/alternatives.log
> var/log/bootstrap.log
> var/log/dpkg.log
> var/log/apt/history.log
> var/log/apt/term.log
> ==============================================

That said and looking at the list ... it all seems harmless. Maybe not
_all_ but a log file or a date here and there can maybe be ignored.

I never really got the idea ... if one wants "exactly" the same result,
there is no reason to rebuild. You just store/distribute the
binary result. But hey you might have your reasons and explain those.

> Steps to check reproducible failures in isar
> ====================================
> $ . isar-init-build-env ../build1 && bitbake
> mc:qemuamd64-buster-tgz:isar-image-base $  . isar-init-build-env
> ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base $ mkdir
> -p rootfs1 rootfs2 $ tar -xzvf
> ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz
> -C ./rootfs1/ $ tar -xzvf
> ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz
> -C ./rootfs2/ $ rsync -nrclv ./rootfs1/ ./rootfs2/ > difference.txt
> ====================================

This is not even remotely close. Here you have been really lucky and
all the "diff" you got was caused by the build. If you introduce a long
pause between the builds ... you will get actually very different
results.
That is a feature .. because isar is tracking debian. But in scenarios
like yours it can be seen as a bug. In which case you need to build
against a custom debian mirror or snapshot.debian.org. (unfortunately
hard because the servers have rate limits, but isar-image-base could
work, or you restart that bitbake a few times)
Snapshot is also a good way to try that ... try a "buster" from a few
months ago for "build1".

Or if you want to track build1 but _not_ track build2 you should use
ISAR_USE_CACHED_BASE_REPO = "1" for that offline rebuild.

> 
> From the reproducible failures I found there are three different
> areas to fix these problem
> 
> 1.       Changelog file generation, which is embedding the build time
> date value at here
> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fdebianize.bbclass%23L34&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=sSB%2BhM1xQauyMIju%2FlJ0GsgdBXYds2C61qpUoQPwuQw%3D&amp;reserved=0
> )

That is a good finding if we want to do something about the "problem".
One could maybe derive the "date" from the file-modification time of
the recipe calling deb_debianize.
But now you have fun with git and will need git-restore-mtime. We could
also force people to put a fixed string there and only call date if
that string is not in place.

> 2.       Log files generated by different application, which are
> adding build time values, I think we can remove these files if it is
> not required after build. ( I tried at here
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fimage.bbclass%23L183&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=kUZoEeqRwUQWk8jZQ%2Bp6t3zO4czouhLQjGKh9RPB8kc%3D&amp;reserved=0
> but it did not work)

Doing it there would be a good place. You could also use
ROOTFS_POSTPROCESS_COMMAND which allows such things for layers so you
do not need to touch the core.

I could envision something like

for f in "find all files not owned by any package":
	if f start.with(/etc)
		continue
  	if other funny exception
		continue
	rm f

In addition to what you want ... this would also shrink that rootfs,
which would be nice even for people that do not care about repro.
logs, tmpdirs, caches would be nice to get rid of.

> 3.       Cache and temporary files, I think we can delete these files
> also.

See previous. Just do all at once asking the package manager which
files it does not know. This will also enforce a really nice discipline
on users to not abuse ROOTFS_POSTPROCESS_COMMAND to smuggle files into
the rootfs.

> Please guide me to fix these issues.

So while i am not 100% with the whole repro idea ... and whether it can
really be done in complex layers ... because you are really not
building a complicated thing here ...

More real use-cases will contain many more packages build by isar,
maybe introducing their own share of "repro" mistakes.
So the "cherry on the cake" would be a helper script to allow anyone to
spot repro diffs. It would run the same build, one online once
ISAR_USE_CACHED_BASE_REPO and spit out two folders and a diff summary.
To give anyone with repro in mind a chance to check their layer. In
fact one has to wonder if such a script should be added to OE or
already exists there.
And one would add that to CI to find new problems as they are
introduced.

I think that allowing to provide a DEBIAN_CHANGELOG_DATE to enforce a
string and not call date could be an interesting patch.

And that a "delete everything not owned by any package" would make a
really nice addition as well.

Both as steps that are on their own valuable and happen to also work in
the direction of reproducible builds.

regards,
Henning

> Thanks,
> Venkata.

Re: [isar] reproducible build failures

[Jan Kiszka]

On 03.09.21 19:10, Henning Schild wrote:
> Hi there,
> 
> Am Fri, 3 Sep 2021 15:19:21 +0000
> schrieb <Venkata.Pyla@toshiba-tsip.com>:
> 
>> Hi,
>>
>> I am using isar system in isar-cip-core project [1] where I found
>> some reproducible failures, which may be good to fix in the isar
>> system. I am not good in modifying the isar system, so could you
>> please guide me to fix these problems?
> 
> Well ... for isar and maybe to some degree also to debian, a truly
> reproducible build would be a new topic that so far has been ignored.
> 

Not completely new, see Debian wiki.

>> Here are the steps to check the reproducible failures in
>> isar-cip-core project:
>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/12
>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/13
>>
>> I also verified the reproducibility in the isar system and found
>> similar failures that are copied below:
>> ============================================== tmp/gpghomefHv8eRhk43/
>> tmp/gpghomefHv8eRhk43/private-keys-v1.d/
>> usr/share/doc/hello/changelog.Debian.gz
>> var/cache/debconf/config.dat
>> var/cache/debconf/config.dat-old
>> var/cache/ldconfig/aux-cache
>> var/lib/dpkg/info/enable-fsck.md5sums
>> var/lib/dpkg/info/example-raw.md5sums
>> var/lib/dpkg/info/hello.md5sums
>> var/lib/dpkg/info/isar-disable-apt-cache.md5sums
>> var/lib/dpkg/info/isar-exclude-docs.md5sums
>> var/lib/dpkg/info/sshd-regen-keys.md5sums
>> var/lib/initramfs-tools/4.19.0-17-amd64
>> var/lib/systemd/catalog/database
>> var/log/alternatives.log
>> var/log/bootstrap.log
>> var/log/dpkg.log
>> var/log/apt/history.log
>> var/log/apt/term.log
>> ==============================================
> 
> That said and looking at the list ... it all seems harmless. Maybe not
> _all_ but a log file or a date here and there can maybe be ignored.
> 
> I never really got the idea ... if one wants "exactly" the same result,
> there is no reason to rebuild. You just store/distribute the
> binary result. But hey you might have your reasons and explain those.

The reasons are the same as for reproducible package build: Validate
that your supply chain, in this case the "last mile", is consistent.

> 
>> Steps to check reproducible failures in isar
>> ====================================
>> $ . isar-init-build-env ../build1 && bitbake
>> mc:qemuamd64-buster-tgz:isar-image-base $  . isar-init-build-env
>> ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base $ mkdir
>> -p rootfs1 rootfs2 $ tar -xzvf
>> ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz
>> -C ./rootfs1/ $ tar -xzvf
>> ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz
>> -C ./rootfs2/ $ rsync -nrclv ./rootfs1/ ./rootfs2/ > difference.txt
>> ====================================
> 
> This is not even remotely close. Here you have been really lucky and
> all the "diff" you got was caused by the build. If you introduce a long
> pause between the builds ... you will get actually very different
> results.
> That is a feature .. because isar is tracking debian. But in scenarios
> like yours it can be seen as a bug. In which case you need to build
> against a custom debian mirror or snapshot.debian.org. (unfortunately
> hard because the servers have rate limits, but isar-image-base could
> work, or you restart that bitbake a few times)
> Snapshot is also a good way to try that ... try a "buster" from a few
> months ago for "build1".
> 
> Or if you want to track build1 but _not_ track build2 you should use
> ISAR_USE_CACHED_BASE_REPO = "1" for that offline rebuild.

Obviously, this only makes sense when building against a snapshot, like
we do in production builds as well, or via the local offline cache.

> 
>>
>> From the reproducible failures I found there are three different
>> areas to fix these problem
>>
>> 1.       Changelog file generation, which is embedding the build time
>> date value at here
>> (https://github.com/ilbers/isar/blob/master/meta/classes/debianize.bbclass#L34
>> )
> 
> That is a good finding if we want to do something about the "problem".
> One could maybe derive the "date" from the file-modification time of
> the recipe calling deb_debianize.
> But now you have fun with git and will need git-restore-mtime. We could
> also force people to put a fixed string there and only call date if
> that string is not in place.
> 
>> 2.       Log files generated by different application, which are
>> adding build time values, I think we can remove these files if it is
>> not required after build. ( I tried at here
>> https://github.com/ilbers/isar/blob/master/meta/classes/image.bbclass#L183
>> but it did not work)
> 
> Doing it there would be a good place. You could also use
> ROOTFS_POSTPROCESS_COMMAND which allows such things for layers so you
> do not need to touch the core.
> 
> I could envision something like
> 
> for f in "find all files not owned by any package":
> 	if f start.with(/etc)
> 		continue
>   	if other funny exception
> 		continue
> 	rm f
> 
> In addition to what you want ... this would also shrink that rootfs,
> which would be nice even for people that do not care about repro.
> logs, tmpdirs, caches would be nice to get rid of.
> 
>> 3.       Cache and temporary files, I think we can delete these files
>> also.
> 
> See previous. Just do all at once asking the package manager which
> files it does not know. This will also enforce a really nice discipline
> on users to not abuse ROOTFS_POSTPROCESS_COMMAND to smuggle files into
> the rootfs.
> 
>> Please guide me to fix these issues.
> 
> So while i am not 100% with the whole repro idea ... and whether it can
> really be done in complex layers ... because you are really not
> building a complicated thing here ...
> 
> More real use-cases will contain many more packages build by isar,
> maybe introducing their own share of "repro" mistakes.
> So the "cherry on the cake" would be a helper script to allow anyone to
> spot repro diffs. It would run the same build, one online once
> ISAR_USE_CACHED_BASE_REPO and spit out two folders and a diff summary.
> To give anyone with repro in mind a chance to check their layer. In
> fact one has to wonder if such a script should be added to OE or
> already exists there.
> And one would add that to CI to find new problems as they are
> introduced.
> 
> I think that allowing to provide a DEBIAN_CHANGELOG_DATE to enforce a
> string and not call date could be an interesting patch.
> 
> And that a "delete everything not owned by any package" would make a
> really nice addition as well.
> 
> Both as steps that are on their own valuable and happen to also work in
> the direction of reproducible builds.
> 

It's a worthwhile goal, no doubt. But it will also be a journey that has
to start somewhere, ideally small. I think we have a couple if nice
first steps here. Adding tooling or at least method descriptions to
detect non-reproducibility in more complex layers will surely be helpful
as well.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [isar] reproducible build failures

[Henning Schild]

Am Fri, 3 Sep 2021 15:19:21 +0000
schrieb <Venkata.Pyla@toshiba-tsip.com>:

> Hi,
> 
> I am using isar system in isar-cip-core project [1] where I found
> some reproducible failures, which may be good to fix in the isar
> system. I am not good in modifying the isar system, so could you
> please guide me to fix these problems?
> 
> Here are the steps to check the reproducible failures in
> isar-cip-core project:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F12&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ofcsE0TXKbwL%2FPUTD2hAEQWvOLXNRFrIvunSOblUAho%3D&amp;reserved=0
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F13&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=M7pZHme2TzX7ZhNbaZW%2BKn26K65ZnKFW4BvnQJldLnw%3D&amp;reserved=0
> 
> I also verified the reproducibility in the isar system and found
> similar failures that are copied below:
> ============================================== tmp/gpghomefHv8eRhk43/
> tmp/gpghomefHv8eRhk43/private-keys-v1.d/
> usr/share/doc/hello/changelog.Debian.gz

I was wondering why only one package is affected here, it should be
more. Turns out that isar-exclude-docs might be a little to thorough.

I sent a patch for that
  [PATCH 3/3] isar-exclude-docs: include "native" package changelogs

And "[PATCH 1/3] debianize: make changelog date reproducible" makes
sure we always use the same date while still allowing user control and
prepending to upstream without "going back in time".

Henning

> var/cache/debconf/config.dat
> var/cache/debconf/config.dat-old
> var/cache/ldconfig/aux-cache
> var/lib/dpkg/info/enable-fsck.md5sums
> var/lib/dpkg/info/example-raw.md5sums
> var/lib/dpkg/info/hello.md5sums
> var/lib/dpkg/info/isar-disable-apt-cache.md5sums
> var/lib/dpkg/info/isar-exclude-docs.md5sums
> var/lib/dpkg/info/sshd-regen-keys.md5sums
> var/lib/initramfs-tools/4.19.0-17-amd64
> var/lib/systemd/catalog/database
> var/log/alternatives.log
> var/log/bootstrap.log
> var/log/dpkg.log
> var/log/apt/history.log
> var/log/apt/term.log
> ==============================================
> 
> Steps to check reproducible failures in isar
> ====================================
> $ . isar-init-build-env ../build1 && bitbake
> mc:qemuamd64-buster-tgz:isar-image-base $  . isar-init-build-env
> ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base $ mkdir
> -p rootfs1 rootfs2 $ tar -xzvf
> ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz
> -C ./rootfs1/ $ tar -xzvf
> ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.tar.gz
> -C ./rootfs2/ $ rsync -nrclv ./rootfs1/ ./rootfs2/ > difference.txt
> ====================================
> 
> From the reproducible failures I found there are three different
> areas to fix these problem
> 
> 1.       Changelog file generation, which is embedding the build time
> date value at here
> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fdebianize.bbclass%23L34&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=sSB%2BhM1xQauyMIju%2FlJ0GsgdBXYds2C61qpUoQPwuQw%3D&amp;reserved=0
> )
> 
> 2.       Log files generated by different application, which are
> adding build time values, I think we can remove these files if it is
> not required after build. ( I tried at here
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fimage.bbclass%23L183&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=kUZoEeqRwUQWk8jZQ%2Bp6t3zO4czouhLQjGKh9RPB8kc%3D&amp;reserved=0
> but it did not work)
> 
> 3.       Cache and temporary files, I think we can delete these files
> also.
> 
> Please guide me to fix these issues.
> 
> Thanks,
> Venkata.

RE: [isar] reproducible build failures

[Venkata.Pyla]

Hi Henning,

Thanks for the below patches

"[PATCH 1/3] debianize: make changelog date reproducible"
"[PATCH 3/3] isar-exclude-docs: include "native" package changelogs"

>-----Original Message-----
>From: Henning Schild <henning.schild@siemens.com>
>Sent: 06 September 2021 14:12
>To: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
>Cc: isar-users@googlegroups.com; jan.kiszka@siemens.com; sangorrin daniel(サ
>ンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <daniel.sangorrin@toshiba.co.jp>; dinesh
>kumar(ＴＳＩＰ) <dinesh.kumar@toshiba-tsip.com>
>Subject: Re: [isar] reproducible build failures
>
>Am Fri, 3 Sep 2021 15:19:21 +0000
>schrieb <Venkata.Pyla@toshiba-tsip.com>:
>
>> Hi,
>>
>> I am using isar system in isar-cip-core project [1] where I found some
>> reproducible failures, which may be good to fix in the isar system. I
>> am not good in modifying the isar system, so could you please guide me
>> to fix these problems?
>>
>> Here are the steps to check the reproducible failures in isar-cip-core
>> project:
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl
>> ab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F12&amp;
>> data=04%7C01%7Cde173c00-e982-4fda-8644-
>47edf4671d63%40ad011.siemens.co
>>
>m%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e14
>95d55
>>
>a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>MC4wLjA
>>
>wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdat
>a=o
>> fcsE0TXKbwL%2FPUTD2hAEQWvOLXNRFrIvunSOblUAho%3D&amp;reserved=0
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl
>> ab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F13&amp;
>> data=04%7C01%7Cde173c00-e982-4fda-8644-
>47edf4671d63%40ad011.siemens.co
>>
>m%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e14
>95d55
>>
>a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
>MC4wLjA
>>
>wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdat
>a=M
>>
>7pZHme2TzX7ZhNbaZW%2BKn26K65ZnKFW4BvnQJldLnw%3D&amp;reserved=0
>>
>> I also verified the reproducibility in the isar system and found
>> similar failures that are copied below:
>> ==============================================
>tmp/gpghomefHv8eRhk43/
>> tmp/gpghomefHv8eRhk43/private-keys-v1.d/
>> usr/share/doc/hello/changelog.Debian.gz
>
>I was wondering why only one package is affected here, it should be more. Turns
>out that isar-exclude-docs might be a little to thorough.
>
>I sent a patch for that
>  [PATCH 3/3] isar-exclude-docs: include "native" package changelogs

Thanks for change, I have tested with this patch and found more changelogs differences in reproducible check,

>
>And "[PATCH 1/3] debianize: make changelog date reproducible" makes sure we
>always use the same date while still allowing user control and prepending to
>upstream without "going back in time".

also this patch fixed the timestamp difference in the changelog and so the reproducible check doesn’t show any difference for the changelogs.

So, these patches will be available in master branch?

Thanks,
Venkata.

>
>Henning
>
>> var/cache/debconf/config.dat
>> var/cache/debconf/config.dat-old
>> var/cache/ldconfig/aux-cache
>> var/lib/dpkg/info/enable-fsck.md5sums
>> var/lib/dpkg/info/example-raw.md5sums
>> var/lib/dpkg/info/hello.md5sums
>> var/lib/dpkg/info/isar-disable-apt-cache.md5sums
>> var/lib/dpkg/info/isar-exclude-docs.md5sums
>> var/lib/dpkg/info/sshd-regen-keys.md5sums
>> var/lib/initramfs-tools/4.19.0-17-amd64
>> var/lib/systemd/catalog/database
>> var/log/alternatives.log
>> var/log/bootstrap.log
>> var/log/dpkg.log
>> var/log/apt/history.log
>> var/log/apt/term.log
>> ==============================================
>>
>> Steps to check reproducible failures in isar
>> ====================================
>> $ . isar-init-build-env ../build1 && bitbake
>> mc:qemuamd64-buster-tgz:isar-image-base $  . isar-init-build-env
>> ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base $ mkdir
>> -p rootfs1 rootfs2 $ tar -xzvf
>> ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qem
>> uamd64.tar.gz
>> -C ./rootfs1/ $ tar -xzvf
>> ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qem
>> uamd64.tar.gz -C ./rootfs2/ $ rsync -nrclv ./rootfs1/ ./rootfs2/ >
>> difference.txt ====================================
>>
>> From the reproducible failures I found there are three different areas
>> to fix these problem
>>
>> 1.       Changelog file generation, which is embedding the build time
>> date value at here
>> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
>>
>hub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fdebianize.b
>> bclass%23L34&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-
>47edf4671d63%4
>>
>0ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95
>794f
>>
>d4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTW
>FpbGZsb3
>>
>d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
>%7
>>
>C2000&amp;sdata=sSB%2BhM1xQauyMIju%2FlJ0GsgdBXYds2C61qpUoQPwuQw
>%3D&amp
>> ;reserved=0
>> )
>>
>> 2.       Log files generated by different application, which are
>> adding build time values, I think we can remove these files if it is
>> not required after build. ( I tried at here
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
>>
>ub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fimage.bbcla
>s
>> s%23L183&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-
>47edf4671d63%40ad0
>>
>11.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794f
>d4ad
>>
>dab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpb
>GZsb3d8ey
>>
>JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
>200
>>
>0&amp;sdata=kUZoEeqRwUQWk8jZQ%2Bp6t3zO4czouhLQjGKh9RPB8kc%3D&a
>mp;reser
>> ved=0
>> but it did not work)
>>
>> 3.       Cache and temporary files, I think we can delete these files
>> also.
>>
>> Please guide me to fix these issues.
>>
>> Thanks,
>> Venkata.

Re: [isar] reproducible build failures

[Henning Schild]

Am Tue, 14 Sep 2021 07:29:07 +0000
schrieb <Venkata.Pyla@toshiba-tsip.com>:

> Hi Henning,
> 
> Thanks for the below patches
> 
> "[PATCH 1/3] debianize: make changelog date reproducible"
> "[PATCH 3/3] isar-exclude-docs: include "native" package changelogs"
> 
> >-----Original Message-----
> >From: Henning Schild <henning.schild@siemens.com>
> >Sent: 06 September 2021 14:12
> >To: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
> >Cc: isar-users@googlegroups.com; jan.kiszka@siemens.com; sangorrin
> >daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ)
> ><daniel.sangorrin@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ)
> ><dinesh.kumar@toshiba-tsip.com> Subject: Re: [isar] reproducible
> >build failures
> >
> >Am Fri, 3 Sep 2021 15:19:21 +0000
> >schrieb <Venkata.Pyla@toshiba-tsip.com>:
> >  
> >> Hi,
> >>
> >> I am using isar system in isar-cip-core project [1] where I found
> >> some reproducible failures, which may be good to fix in the isar
> >> system. I am not good in modifying the isar system, so could you
> >> please guide me to fix these problems?
> >>
> >> Here are the steps to check the reproducible failures in
> >> isar-cip-core project:
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl
> >> ab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F12&amp;
> >> data=04%7C01%7Cde173c00-e982-4fda-8644-  
> >47edf4671d63%40ad011.siemens.co  
> >>  
> >m%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e14
> >95d55  
> >>  
> >a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> >MC4wLjA  
> >>  
> >wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdat
> >a=o  
> >> fcsE0TXKbwL%2FPUTD2hAEQWvOLXNRFrIvunSOblUAho%3D&amp;reserved=0
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl
> >> ab.com%2Fcip-project%2Fcip-core%2Fisar-cip-core%2F-%2Fissues%2F13&amp;
> >> data=04%7C01%7Cde173c00-e982-4fda-8644-  
> >47edf4671d63%40ad011.siemens.co  
> >>  
> >m%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794fd4addab42e14
> >95d55  
> >>  
> >a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> >MC4wLjA  
> >>  
> >wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdat
> >a=M  
> >>  
> >7pZHme2TzX7ZhNbaZW%2BKn26K65ZnKFW4BvnQJldLnw%3D&amp;reserved=0  
> >>
> >> I also verified the reproducibility in the isar system and found
> >> similar failures that are copied below:
> >> ==============================================  
> >tmp/gpghomefHv8eRhk43/  
> >> tmp/gpghomefHv8eRhk43/private-keys-v1.d/
> >> usr/share/doc/hello/changelog.Debian.gz  
> >
> >I was wondering why only one package is affected here, it should be
> >more. Turns out that isar-exclude-docs might be a little to thorough.
> >
> >I sent a patch for that
> >  [PATCH 3/3] isar-exclude-docs: include "native" package changelogs
> >  
> 
> Thanks for change, I have tested with this patch and found more
> changelogs differences in reproducible check,
> 
> >
> >And "[PATCH 1/3] debianize: make changelog date reproducible" makes
> >sure we always use the same date while still allowing user control
> >and prepending to upstream without "going back in time".  
> 
> also this patch fixed the timestamp difference in the changelog and
> so the reproducible check doesn’t show any difference for the
> changelogs.
> 
> So, these patches will be available in master branch?

In isar things are usually merged to next and master follows pretty
slowly. But next is never rebased so it is safe to use as well.

So far the patches are on the list for review. If no review comments
suggest that change is needed, they will eventually be picked up.

You might be able to speed that up by giving explicit positive review
feedback like you did here.

But i guess we will just wait and and the meantime you can apply those
patches locally if you care about them a lot.

Henning

> Thanks,
> Venkata.
> 
> >
> >Henning
> >  
> >> var/cache/debconf/config.dat
> >> var/cache/debconf/config.dat-old
> >> var/cache/ldconfig/aux-cache
> >> var/lib/dpkg/info/enable-fsck.md5sums
> >> var/lib/dpkg/info/example-raw.md5sums
> >> var/lib/dpkg/info/hello.md5sums
> >> var/lib/dpkg/info/isar-disable-apt-cache.md5sums
> >> var/lib/dpkg/info/isar-exclude-docs.md5sums
> >> var/lib/dpkg/info/sshd-regen-keys.md5sums
> >> var/lib/initramfs-tools/4.19.0-17-amd64
> >> var/lib/systemd/catalog/database
> >> var/log/alternatives.log
> >> var/log/bootstrap.log
> >> var/log/dpkg.log
> >> var/log/apt/history.log
> >> var/log/apt/term.log
> >> ==============================================
> >>
> >> Steps to check reproducible failures in isar
> >> ====================================
> >> $ . isar-init-build-env ../build1 && bitbake
> >> mc:qemuamd64-buster-tgz:isar-image-base $  . isar-init-build-env
> >> ../build2 && bitbake mc:qemuamd64-buster-tgz:isar-image-base $
> >> mkdir -p rootfs1 rootfs2 $ tar -xzvf
> >> ./build1/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qem
> >> uamd64.tar.gz
> >> -C ./rootfs1/ $ tar -xzvf
> >> ./build2/tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qem
> >> uamd64.tar.gz -C ./rootfs2/ $ rsync -nrclv ./rootfs1/ ./rootfs2/ >
> >> difference.txt ====================================
> >>
> >> From the reproducible failures I found there are three different
> >> areas to fix these problem
> >>
> >> 1.       Changelog file generation, which is embedding the build
> >> time date value at here
> >> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> >>  
> >hub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fdebianize.b
> > 
> >> bclass%23L34&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-  
> >47edf4671d63%4  
> >>  
> >0ad011.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95
> >794f  
> >>  
> >d4addab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTW
> >FpbGZsb3  
> >>  
> >d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> >%7  
> >>  
> >C2000&amp;sdata=sSB%2BhM1xQauyMIju%2FlJ0GsgdBXYds2C61qpUoQPwuQw
> >%3D&amp  
> >> ;reserved=0
> >> )
> >>
> >> 2.       Log files generated by different application, which are
> >> adding build time values, I think we can remove these files if it
> >> is not required after build. ( I tried at here
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> >>  
> >ub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fmeta%2Fclasses%2Fimage.bbcla
> >s  
> >> s%23L183&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-  
> >47edf4671d63%40ad0  
> >>  
> >11.siemens.com%7C186fea861bb04bae174708d96eee3bfb%7C38ae3bcd95794f
> >d4ad  
> >>  
> >dab42e1495d55a%7C1%7C0%7C637662792787877923%7CUnknown%7CTWFpb
> >GZsb3d8ey  
> >>  
> >JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> >200  
> >>  
> >0&amp;sdata=kUZoEeqRwUQWk8jZQ%2Bp6t3zO4czouhLQjGKh9RPB8kc%3D&a
> >mp;reser  
> >> ved=0
> >> but it did not work)
> >>
> >> 3.       Cache and temporary files, I think we can delete these
> >> files also.
> >>
> >> Please guide me to fix these issues.
> >>
> >> Thanks,
> >> Venkata.  
>