[PATCH] classes/repository: use the proper filename to find the packages under repo

[PATCH] classes/repository: use the proper filename to find the packages under repo

[srinuvasan.a via isar-users]

From: srinuvasan <srinuvasan.a@siemens.com>

The very first time when we generate the base-apt the REPO_BASE_DIR
is empty, it doesn't contain any packages, hence repo_contains_package
function returns 2, based on the return value (2), we are adding all the
packages to the empty repo by calling the repo_add_packages function.

After clearing the temporary and cache files, when we retrigger the base-apt
we should skip the repo_add_packages function for all the packages (assume that
repo packages are not contaminated), but we observed repo_contains_function
says some of the packages not available in the existing repo, but really those
packages available in the repo.

Here the issue is, reprepro caching all the packages by skipping the
epoch version if any packages have, and all the packages contain only the
<upstream_version>-<debian_version>.

But in the download folder , few packages contain the epoch version,
f.e: automake, git-man, ssh, now we try to find the packages with
epoch version in APT repo, but APT repos skip the epoch version
during repo generation.

With that wrong finding again we are calling the repo_add_package
function for few packages, even those packages already available in the repo.

Skip the epoch version during finding the packages in the repo, with that we can eliminate
unnecessarily adding the packages to repo.

Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
---
 meta/classes/repository.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 28e712fd..7379658d 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -99,7 +99,8 @@ repo_contains_package() {
     local file="$2"
     local package
 
-    package=$(find ${dir} -name ${file##*/})
+    file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
+    package=$(find ${dir} -name ${file_name})
     if [ -n "$package" ]; then
         # yes
         cmp --silent "$package" "$file" && return 0
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241116172900.897898-1-srinuvasan.a%40siemens.com.

Re: [PATCH] classes/repository: use the proper filename to find the packages under repo

['cedric.hombourger@siemens.com' via isar-users]

Hello Srinu,

I have a different implementation of this function that I was going to
submit after doing some clean-up and running a full CI build

Here's anyhow what it looks like:

# lookup ${file} in the database for the current suite
package=$(reprepro -b ${dir} --dbdir ${dbdir} \
                   --list-format 'Filename\n' \
                   listfilter ${distro} \
                   'Filename (% /'${file##*/}')' | head -n 1)
if [ -n "$package" ]; then
    # yes
    cmp --silent "$package" "$file" && return 0

    # yes but not the exact same file
    return 1
fi
# no
return 2

I would suggest you come up with a more concise description of your
change and ideally some way to reproduce and exhibit the problem as I
am not sure I follow your problem statement

On Sat, 2024-11-16 at 22:58 +0530, srinuvasan.a@siemens.com wrote:
> From: srinuvasan <srinuvasan.a@siemens.com>
> 
> The very first time when we generate the base-apt the REPO_BASE_DIR
> is empty, it doesn't contain any packages, hence
> repo_contains_package
> function returns 2, based on the return value (2), we are adding all
> the
> packages to the empty repo by calling the repo_add_packages function.
> 
> After clearing the temporary and cache files, when we retrigger the
> base-apt
> we should skip the repo_add_packages function for all the packages
> (assume that
> repo packages are not contaminated), but we observed
> repo_contains_function
> says some of the packages not available in the existing repo, but
> really those
> packages available in the repo.
> 
> Here the issue is, reprepro caching all the packages by skipping the
> epoch version if any packages have, and all the packages contain only
> the
> <upstream_version>-<debian_version>.
> 
> But in the download folder , few packages contain the epoch version,
> f.e: automake, git-man, ssh, now we try to find the packages with
> epoch version in APT repo, but APT repos skip the epoch version
> during repo generation.
> 
> With that wrong finding again we are calling the repo_add_package
> function for few packages, even those packages already available in
> the repo.
> 
> Skip the epoch version during finding the packages in the repo, with
> that we can eliminate
> unnecessarily adding the packages to repo.
> 
> Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> ---
>  meta/classes/repository.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/repository.bbclass
> b/meta/classes/repository.bbclass
> index 28e712fd..7379658d 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -99,7 +99,8 @@ repo_contains_package() {
>      local file="$2"
>      local package
>  
> -    package=$(find ${dir} -name ${file##*/})
> +    file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7753363cffbf014661ddfe439c96417a971a9f3c.camel%40siemens.com.

Re: [PATCH] classes/repository: use the proper filename to find the packages under repo

['cedric.hombourger@siemens.com' via isar-users]

On Sat, 2024-11-16 at 22:58 +0530, srinuvasan.a@siemens.com wrote:
> From: srinuvasan <srinuvasan.a@siemens.com>
> 
> The very first time when we generate the base-apt the REPO_BASE_DIR
> is empty, it doesn't contain any packages, hence
> repo_contains_package
> function returns 2, based on the return value (2), we are adding all
> the
> packages to the empty repo by calling the repo_add_packages function.
> 
> After clearing the temporary and cache files, when we retrigger the
> base-apt
> we should skip the repo_add_packages function for all the packages
> (assume that
> repo packages are not contaminated), but we observed
> repo_contains_function
> says some of the packages not available in the existing repo, but
> really those
> packages available in the repo.
> 
> Here the issue is, reprepro caching all the packages by skipping the
> epoch version if any packages have, and all the packages contain only
> the
> <upstream_version>-<debian_version>.
> 
> But in the download folder , few packages contain the epoch version,
> f.e: automake, git-man, ssh, now we try to find the packages with
> epoch version in APT repo, but APT repos skip the epoch version
> during repo generation.

As I was not fully understanding the problem, I checked the packages
you listed and now it is much clearer:

$ find build/base-apt/local/ build/base-apt/downloads/ -name git-man\*
build/base-apt/local/apt/debian/pool/main/g/git/git-man_2.39.5-
0+deb12u1_all.deb
build/base-apt/downloads/deb/debian-bookworm/git-man_1%3a2.39.5-
0+deb12u1_all.deb

Could you reword your commit message to be more concise?

> 
> With that wrong finding again we are calling the repo_add_package
> function for few packages, even those packages already available in
> the repo.
> 
> Skip the epoch version during finding the packages in the repo, with
> that we can eliminate
> unnecessarily adding the packages to repo.
> 
> Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> ---
>  meta/classes/repository.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/repository.bbclass
> b/meta/classes/repository.bbclass
> index 28e712fd..7379658d 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -99,7 +99,8 @@ repo_contains_package() {
>      local file="$2"
>      local package
>  
> -    package=$(find ${dir} -name ${file##*/})
> +    file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/2e8f538fc2a14f8f235819e33656fb6e2a850012.camel%40siemens.com.

RE: [PATCH] classes/repository: use the proper filename to find the packages under repo

['Arjunan, Srinu' via isar-users]

Thanks cedric for the review, I will refactor the commit message even more concise.

Many thanks,
Srinu

-----Original Message-----
From: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com> 
Sent: 17 November 2024 05:24
To: isar-users@googlegroups.com; Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>
Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>
Subject: Re: [PATCH] classes/repository: use the proper filename to find the packages under repo

On Sat, 2024-11-16 at 22:58 +0530, srinuvasan.a@siemens.com wrote:
> From: srinuvasan <srinuvasan.a@siemens.com>
> 
> The very first time when we generate the base-apt the REPO_BASE_DIR is 
> empty, it doesn't contain any packages, hence repo_contains_package 
> function returns 2, based on the return value (2), we are adding all 
> the packages to the empty repo by calling the repo_add_packages 
> function.
> 
> After clearing the temporary and cache files, when we retrigger the 
> base-apt we should skip the repo_add_packages function for all the 
> packages (assume that repo packages are not contaminated), but we 
> observed repo_contains_function says some of the packages not 
> available in the existing repo, but really those packages available in 
> the repo.
> 
> Here the issue is, reprepro caching all the packages by skipping the 
> epoch version if any packages have, and all the packages contain only 
> the <upstream_version>-<debian_version>.
> 
> But in the download folder , few packages contain the epoch version,
> f.e: automake, git-man, ssh, now we try to find the packages with 
> epoch version in APT repo, but APT repos skip the epoch version during 
> repo generation.

As I was not fully understanding the problem, I checked the packages you listed and now it is much clearer:

$ find build/base-apt/local/ build/base-apt/downloads/ -name git-man\*
build/base-apt/local/apt/debian/pool/main/g/git/git-man_2.39.5-
0+deb12u1_all.deb
build/base-apt/downloads/deb/debian-bookworm/git-man_1%3a2.39.5-
0+deb12u1_all.deb

Could you reword your commit message to be more concise?

> 
> With that wrong finding again we are calling the repo_add_package 
> function for few packages, even those packages already available in 
> the repo.
> 
> Skip the epoch version during finding the packages in the repo, with 
> that we can eliminate unnecessarily adding the packages to repo.
> 
> Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> ---
>  meta/classes/repository.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/repository.bbclass 
> b/meta/classes/repository.bbclass index 28e712fd..7379658d 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -99,7 +99,8 @@ repo_contains_package() {
>      local file="$2"
>      local package
>  
> -    package=$(find ${dir} -name ${file##*/})
> +    file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0

--
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/TY0PR06MB57328D43B6C0320B1C9CC7C788262%40TY0PR06MB5732.apcprd06.prod.outlook.com.

[PATCH v2] classes/repository: use the proper filename to find the packages under repo

[srinuvasan.a via isar-users]

From: srinuvasan <srinuvasan.a@siemens.com>

This patch fixes the issue in repo_contians_package function, here we
are adding a few packages to repo again even though the repo already contains those
packages, ideally we should skip those

reproduce the issue:

1. Create the base-apt
2. Again retrigger the base-apt (assume that there is no changes in the previously built base-apt)

here we observed few packages getting added again to the APT, even those
packages already available in the repo.

Basically few packages having the epoch version (automake, git-man, ssh) in the download folder,
but the same packages are cached in the repo without the epoch version, due to this
mismatch, it tries to add the packages again into the repo, to fix this issue,
just find the package name in the repo without the epoch version.

Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
---
 meta/classes/repository.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 28e712fd..7379658d 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -99,7 +99,8 @@ repo_contains_package() {
     local file="$2"
     local package
 
-    package=$(find ${dir} -name ${file##*/})
+    file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')
+    package=$(find ${dir} -name ${file_name})
     if [ -n "$package" ]; then
         # yes
         cmp --silent "$package" "$file" && return 0
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241117154515.906833-1-srinuvasan.a%40siemens.com.

Re: [PATCH v2] classes/repository: use the proper filename to find the packages under repo

[Uladzimir Bely]

On Sun, 2024-11-17 at 21:15 +0530, srinuvasan.a via isar-users wrote:
> From: srinuvasan <srinuvasan.a@siemens.com>
> 
> This patch fixes the issue in repo_contians_package function, here we
> are adding a few packages to repo again even though the repo already
> contains those
> packages, ideally we should skip those
> 
> reproduce the issue:
> 
> 1. Create the base-apt
> 2. Again retrigger the base-apt (assume that there is no changes in
> the previously built base-apt)
> 
> here we observed few packages getting added again to the APT, even
> those
> packages already available in the repo.
> 
> Basically few packages having the epoch version (automake, git-man,
> ssh) in the download folder,
> but the same packages are cached in the repo without the epoch
> version, due to this
> mismatch, it tries to add the packages again into the repo, to fix
> this issue,
> just find the package name in the repo without the epoch version.
> 
> Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> ---
>  meta/classes/repository.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/repository.bbclass
> b/meta/classes/repository.bbclass
> index 28e712fd..7379658d 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -99,7 +99,8 @@ repo_contains_package() {
>      local file="$2"
>      local package
>  
> -    package=$(find ${dir} -name ${file##*/})
> +    file_name=$(echo "${file##*/}" | sed 's/[0-9]%3a//g')

Not sure if epoch versions more than 9 exist for some packages, but I
would add '\+'   here to consider big epoch numbers:

```
$ echo git-man_1%3a2.39.5-0+deb12u1_all.deb | sed 's/[0-9]%3a//g'
git-man_2.39.5-0+deb12u1_all.deb
$ echo git-man_10%3a2.39.5-0+deb12u1_all.deb | sed 's/[0-9]%3a//g'
git-man_12.39.5-0+deb12u1_all.deb
$ echo git-man_10%3a2.39.5-0+deb12u1_all.deb | sed 's/[0-9]\+%3a//g'
git-man_2.39.5-0+deb12u1_all.deb
```

> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0
> -- 
> 2.34.1
> 

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/47c5e76990ba95e85e07df4542d3aa74e6e002f3.camel%40ilbers.de.

[PATCH v3] classes/repository: use the proper filename to find the packages under repo

[srinuvasan.a via isar-users]

From: Srinuvasan A <srinuvasan.a@siemens.com>

This patch fixes the issue in repo_contians_package function, here we
are adding a few packages to repo again even though the repo already contains those
packages, ideally we should skip those

reproduce the issue:

1. Create the base-apt
2. Again retrigger the base-apt (assume that there is no changes in the previously built base-apt)

here we observed few packages getting added again to the APT, even those
packages already available in the repo.

Basically few packages having the epoch version (automake, git-man, ssh) in the download folder,
but the same packages are cached in the repo without the epoch version, due to this
mismatch, it tries to add the packages again into the repo, to fix this issue,
just find the package name in the repo without the epoch version.

Signed-off-by: Srinuvasan A <srinuvasan.a@siemens.com>
---
 meta/classes/repository.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 28e712fd..043c8621 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -99,7 +99,8 @@ repo_contains_package() {
     local file="$2"
     local package
 
-    package=$(find ${dir} -name ${file##*/})
+    file_name=$(echo "${file##*/}" | sed 's/[0-9]\+%3a//g')
+    package=$(find ${dir} -name ${file_name})
     if [ -n "$package" ]; then
         # yes
         cmp --silent "$package" "$file" && return 0
-- 
2.30.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241120093204.977261-1-srinuvasan.a%40siemens.com.

Re: [PATCH v3] classes/repository: use the proper filename to find the packages under repo

[Uladzimir Bely]

On Wed, 2024-11-20 at 15:02 +0530, srinuvasan.a@siemens.com wrote:
> From: Srinuvasan A <srinuvasan.a@siemens.com>
> 
> This patch fixes the issue in repo_contians_package function, here we
> are adding a few packages to repo again even though the repo already
> contains those
> packages, ideally we should skip those
> 
> reproduce the issue:
> 
> 1. Create the base-apt
> 2. Again retrigger the base-apt (assume that there is no changes in
> the previously built base-apt)
> 
> here we observed few packages getting added again to the APT, even
> those
> packages already available in the repo.
> 
> Basically few packages having the epoch version (automake, git-man,
> ssh) in the download folder,
> but the same packages are cached in the repo without the epoch
> version, due to this
> mismatch, it tries to add the packages again into the repo, to fix
> this issue,
> just find the package name in the repo without the epoch version.
> 
> Signed-off-by: Srinuvasan A <srinuvasan.a@siemens.com>

Hello.

Since now we have https://groups.google.com/g/isar-users/c/NrVHusKyNTA
series merged to next, this patch gets incompatible and likely not
needed anymore.

> ---
>  meta/classes/repository.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/repository.bbclass
> b/meta/classes/repository.bbclass
> index 28e712fd..043c8621 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -99,7 +99,8 @@ repo_contains_package() {
>      local file="$2"
>      local package
>  
> -    package=$(find ${dir} -name ${file##*/})
> +    file_name=$(echo "${file##*/}" | sed 's/[0-9]\+%3a//g')
> +    package=$(find ${dir} -name ${file_name})
>      if [ -n "$package" ]; then
>          # yes
>          cmp --silent "$package" "$file" && return 0

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7930cd884bc90928657fcf4628cb500a6d20f5ec.camel%40ilbers.de.