[PATCH 0/3] Switch to own isar-docker image

[PATCH 0/3] Switch to own isar-docker image

[Uladzimir Bely]

In comparison with previously used kas-isar image, switch to own
isar-docker one, that better fit CI needs:
 - Avocado testsuite preinstalled;
 - Qemu-system packages for run tests preinstalled;
 - Potentially, avocado can be run directly, without ci_build.sh.

Uladzimir Bely (3):
  testsuite: Dockerfile for isar-docker image
  testsuite: Add README for isar-docker
  gitlab-ci: Switch to own isar-docker image

 .gitlab-ci.yml                            |   3 +-
 testsuite/dockerdata/.dockerignore        |   3 +
 testsuite/dockerdata/Dockerfile           |  48 ++++++
 testsuite/dockerdata/README.md            |  19 +++
 testsuite/dockerdata/container-entrypoint |   9 ++
 testsuite/dockerdata/contrib/oe-git-proxy | 187 ++++++++++++++++++++++
 6 files changed, 268 insertions(+), 1 deletion(-)
 create mode 100644 testsuite/dockerdata/.dockerignore
 create mode 100644 testsuite/dockerdata/Dockerfile
 create mode 100644 testsuite/dockerdata/README.md
 create mode 100755 testsuite/dockerdata/container-entrypoint
 create mode 100755 testsuite/dockerdata/contrib/oe-git-proxy

-- 
2.20.1

[PATCH 1/3] testsuite: Dockerfile for isar-docker image

[Uladzimir Bely]

Add dockerfile and related files for generating isar-docker image
supposed to be used in docker-based CI systems.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/dockerdata/.dockerignore        |   2 +
 testsuite/dockerdata/Dockerfile           |  48 ++++++
 testsuite/dockerdata/container-entrypoint |   9 ++
 testsuite/dockerdata/contrib/oe-git-proxy | 187 ++++++++++++++++++++++
 4 files changed, 246 insertions(+)
 create mode 100644 testsuite/dockerdata/.dockerignore
 create mode 100644 testsuite/dockerdata/Dockerfile
 create mode 100755 testsuite/dockerdata/container-entrypoint
 create mode 100755 testsuite/dockerdata/contrib/oe-git-proxy

diff --git a/testsuite/dockerdata/.dockerignore b/testsuite/dockerdata/.dockerignore
new file mode 100644
index 00000000..6e19512a
--- /dev/null
+++ b/testsuite/dockerdata/.dockerignore
@@ -0,0 +1,2 @@
+.dockerignore
+Dockerfile
diff --git a/testsuite/dockerdata/Dockerfile b/testsuite/dockerdata/Dockerfile
new file mode 100644
index 00000000..7372c752
--- /dev/null
+++ b/testsuite/dockerdata/Dockerfile
@@ -0,0 +1,48 @@
+FROM debian:bullseye-slim as isar-docker
+
+ARG TARGETPLATFORM
+ARG DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && \
+    apt-get install -y locales && \
+    localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
+
+ENV LANG=en_US.utf8
+ENV LC_ALL=en_US.UTF-8
+
+# Isar main dependencies
+RUN apt-get install -y -f --no-install-recommends \
+        binfmt-support bzip2 debootstrap dosfstools dpkg-dev gettext-base \
+        git mtools parted python3 quilt qemu-user-static reprepro sudo \
+        unzip xz-utils git-buildpackage pristine-tar sbuild schroot zstd \
+        python3-distutils \
+        umoci skopeo
+
+# Isar testsuite dependencies
+RUN apt-get install --no-install-recommends -y \
+        python3-pip && \
+    pip3 --proxy=$https_proxy install avocado-framework==100.1 && \
+    rm -rf $(pip3 cache dir) && \
+    apt-get install -y -f --no-install-recommends \
+        qemu-system ovmf
+
+RUN apt-get clean && \
+    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* 
+
+COPY contrib/oe-git-proxy /usr/bin/
+ENV GIT_PROXY_COMMAND="oe-git-proxy" \
+    NO_PROXY="*"
+
+RUN echo "builder ALL=NOPASSWD: ALL" > /etc/sudoers.d/builder-nopasswd && \
+    chmod 660 /etc/sudoers.d/builder-nopasswd && \
+    echo "Defaults env_keep += \"ftp_proxy http_proxy https_proxy no_proxy\"" \
+    > /etc/sudoers.d/env_keep && chmod 660 /etc/sudoers.d/env_keep
+
+RUN useradd builder --user-group --create-home --home-dir /builder && \
+    sbuild-adduser builder >/dev/null 2>/dev/null
+
+COPY container-entrypoint /
+
+USER builder
+
+ENTRYPOINT ["/container-entrypoint"]
diff --git a/testsuite/dockerdata/container-entrypoint b/testsuite/dockerdata/container-entrypoint
new file mode 100755
index 00000000..6af07932
--- /dev/null
+++ b/testsuite/dockerdata/container-entrypoint
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+sudo update-binfmts --enable && [ -f /proc/sys/fs/binfmt_misc/status ]
+
+if [ -n "$1" ]; then
+    exec "$@"
+else
+    exec bash
+fi
diff --git a/testsuite/dockerdata/contrib/oe-git-proxy b/testsuite/dockerdata/contrib/oe-git-proxy
new file mode 100755
index 00000000..aa9b9dc9
--- /dev/null
+++ b/testsuite/dockerdata/contrib/oe-git-proxy
@@ -0,0 +1,187 @@
+#!/bin/bash
+
+# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses socat
+# to make SOCKS5 or HTTPS proxy connections.
+# It uses ALL_PROXY or all_proxy or http_proxy to determine the proxy server,
+# protocol, and port.
+# It uses NO_PROXY to skip using the proxy for a comma delimited list of
+# hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It
+# is known to work with both bash and dash shells.
+#
+# Example ALL_PROXY values:
+# ALL_PROXY=socks://socks.example.com:1080
+# ALL_PROXY=https://proxy.example.com:8080
+#
+# Copyright (c) 2013, Intel Corporation.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# AUTHORS
+# Darren Hart <dvhart@linux.intel.com>
+
+# disable pathname expansion, NO_PROXY fields could start with "*" or be it
+set -f
+
+if [ $# -lt 2 -o "$1" = '--help' -o "$1" = '-h' ] ; then
+    echo 'oe-git-proxy: error: the following arguments are required: host port'
+    echo 'Usage: oe-git-proxy host port'
+    echo ''
+    echo 'OpenEmbedded git-proxy - a simple tool to be used via GIT_PROXY_COMMAND.'
+    echo 'It uses socat to make SOCKS or HTTPS proxy connections.'
+    echo 'It uses ALL_PROXY to determine the proxy server, protocol, and port.'
+    echo 'It uses NO_PROXY to skip using the proxy for a comma delimited list'
+    echo 'of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24).'
+    echo 'It is known to work with both bash and dash shells.runs native tools'
+    echo ''
+    echo 'arguments:'
+    echo '  host                proxy host to use'
+    echo '  port                proxy port to use'
+    echo ''
+    echo 'options:'
+    echo '  -h, --help          show this help message and exit'
+    echo ''
+    exit 2
+fi
+
+# Locate the netcat binary
+if [ -z "$SOCAT" ]; then
+	SOCAT=$(which socat 2>/dev/null)
+	if [ $? -ne 0 ]; then
+		echo "ERROR: socat binary not in PATH" 1>&2
+		exit 1
+	fi
+fi
+METHOD=""
+
+# Test for a valid IPV4 quad with optional bitmask
+valid_ipv4() {
+	echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$"
+	return $?
+}
+
+# Convert an IPV4 address into a 32bit integer
+ipv4_val() {
+	IP="$1"
+	SHIFT=24
+	VAL=0
+	for B in $( echo "$IP" | tr '.' ' ' ); do
+		VAL=$(($VAL+$(($B<<$SHIFT))))
+		SHIFT=$(($SHIFT-8))
+	done
+	echo "$VAL"
+}
+
+# Determine if two IPs are equivalent, or if the CIDR contains the IP
+match_ipv4() {
+	CIDR=$1
+	IP=$2
+
+	if [ -z "${IP%%$CIDR}" ]; then
+		return 0
+	fi
+
+	# Determine the mask bitlength
+	BITS=${CIDR##*/}
+	[ "$BITS" != "$CIDR" ] || BITS=32
+	if [ -z "$BITS" ]; then
+		return 1
+	fi
+
+	IPVAL=$(ipv4_val $IP)
+	IP2VAL=$(ipv4_val ${CIDR%%/*})
+
+	# OR in the unmasked bits
+	for i in $(seq 0 $((32-$BITS))); do
+		IP2VAL=$(($IP2VAL|$((1<<$i))))
+		IPVAL=$(($IPVAL|$((1<<$i))))
+	done
+
+	if [ $IPVAL -eq $IP2VAL ]; then
+		return 0
+	fi
+	return 1
+}
+
+# Test to see if GLOB matches HOST
+match_host() {
+	HOST=$1
+	GLOB=$2
+
+	if [ -z "${HOST%%*$GLOB}" ]; then
+		return 0
+	fi
+
+	# Match by netmask
+	if valid_ipv4 $GLOB; then
+		for HOST_IP in $(getent ahostsv4 $HOST | grep ' STREAM ' | cut -d ' ' -f 1) ; do
+			if valid_ipv4 $HOST_IP; then
+				match_ipv4 $GLOB $HOST_IP
+				if [ $? -eq 0 ]; then
+					return 0
+				fi
+			fi
+		done
+	fi
+
+	return 1
+}
+
+# If no proxy is set or needed, just connect directly
+METHOD="TCP:$1:$2"
+
+[ -z "${ALL_PROXY}" ] && ALL_PROXY=$all_proxy
+[ -z "${ALL_PROXY}" ] && ALL_PROXY=$http_proxy
+
+if [ -z "$ALL_PROXY" ]; then
+	exec $SOCAT STDIO $METHOD
+fi
+
+# Connect directly to hosts in NO_PROXY
+for H in $( echo "$NO_PROXY" | tr ',' ' ' ); do
+	if match_host $1 $H; then
+		exec $SOCAT STDIO $METHOD
+	fi
+done
+
+# Proxy is necessary, determine protocol, server, and port
+# extract protocol
+PROTO=${ALL_PROXY%://*}
+# strip protocol:// from string
+ALL_PROXY=${ALL_PROXY#*://}
+# extract host & port parts:
+#   1) drop username/password
+PROXY=${ALL_PROXY##*@}
+#   2) remove optional trailing /?
+PROXY=${PROXY%%/*}
+#   3) extract optional port
+PORT=${PROXY##*:}
+if [ "$PORT" = "$PROXY" ]; then
+	PORT=""
+fi
+#   4) remove port
+PROXY=${PROXY%%:*}
+
+# extract username & password
+PROXYAUTH="${ALL_PROXY%@*}"
+[ "$PROXYAUTH" = "$ALL_PROXY" ] && PROXYAUTH=
+[ -n "${PROXYAUTH}" ] && PROXYAUTH=",proxyauth=${PROXYAUTH}"
+
+if [ "$PROTO" = "socks" ] || [ "$PROTO" = "socks4a" ]; then
+	if [ -z "$PORT" ]; then
+		PORT="1080"
+	fi
+	METHOD="SOCKS4A:$PROXY:$1:$2,socksport=$PORT"
+elif [ "$PROTO" = "socks4" ]; then
+	if [ -z "$PORT" ]; then
+		PORT="1080"
+	fi
+	METHOD="SOCKS4:$PROXY:$1:$2,socksport=$PORT"
+else
+	# Assume PROXY (http, https, etc)
+	if [ -z "$PORT" ]; then
+		PORT="8080"
+	fi
+	METHOD="PROXY:$PROXY:$1:$2,proxyport=${PORT}${PROXYAUTH}"
+fi
+
+exec $SOCAT STDIO "$METHOD"
-- 
2.20.1

[PATCH 2/3] testsuite: Add README for isar-docker

[Uladzimir Bely]

Document isar-docker image prepare and publish steps

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/dockerdata/.dockerignore |  1 +
 testsuite/dockerdata/README.md     | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 testsuite/dockerdata/README.md

diff --git a/testsuite/dockerdata/.dockerignore b/testsuite/dockerdata/.dockerignore
index 6e19512a..d06a06ab 100644
--- a/testsuite/dockerdata/.dockerignore
+++ b/testsuite/dockerdata/.dockerignore
@@ -1,2 +1,3 @@
 .dockerignore
 Dockerfile
+README.md
diff --git a/testsuite/dockerdata/README.md b/testsuite/dockerdata/README.md
new file mode 100644
index 00000000..aae9ddb7
--- /dev/null
+++ b/testsuite/dockerdata/README.md
@@ -0,0 +1,19 @@
+# Creating image
+
+```
+export version="0.1"
+cd <isar_dir>/testsuite/dockerdata
+docker build -t ghcr.io/ilbers/docker-isar:${version} .
+```
+
+# Pushing the image to docker hub
+
+- Configure github token (classic) with `write:packages` permissions.
+
+- Use it for uploading docker image:
+
+```
+docker push ghcr.io/ilbers/docker-isar:${version}
+```
+
+- Make the uploaded package public 
-- 
2.20.1

[PATCH 3/3] gitlab-ci: Switch to own isar-docker image

[Uladzimir Bely]

Instead of using ghcr.io/siemens/kas/kas-isar image for CI, use the own
minimal one including everything required to run Isar testsuite.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .gitlab-ci.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index eb38be54..1682692d 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,4 +1,5 @@
-image: ghcr.io/siemens/kas/kas-isar:3.0.2
+default:
+  image: ghcr.io/ilbers/docker-isar:0.1
 
 variables:
   GIT_STRATEGY: clone
-- 
2.20.1

Re: [PATCH 1/3] testsuite: Dockerfile for isar-docker image

[Schaffner, Tobias]

On 07.03.23 05:36, Uladzimir Bely wrote:
> Add dockerfile and related files for generating isar-docker image
> supposed to be used in docker-based CI systems.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>   testsuite/dockerdata/.dockerignore        |   2 +
>   testsuite/dockerdata/Dockerfile           |  48 ++++++
>   testsuite/dockerdata/container-entrypoint |   9 ++
>   testsuite/dockerdata/contrib/oe-git-proxy | 187 ++++++++++++++++++++++
>   4 files changed, 246 insertions(+)
>   create mode 100644 testsuite/dockerdata/.dockerignore
>   create mode 100644 testsuite/dockerdata/Dockerfile
>   create mode 100755 testsuite/dockerdata/container-entrypoint
>   create mode 100755 testsuite/dockerdata/contrib/oe-git-proxy
> 
> diff --git a/testsuite/dockerdata/.dockerignore b/testsuite/dockerdata/.dockerignore
> new file mode 100644
> index 00000000..6e19512a
> --- /dev/null
> +++ b/testsuite/dockerdata/.dockerignore
> @@ -0,0 +1,2 @@
> +.dockerignore
> +Dockerfile
> diff --git a/testsuite/dockerdata/Dockerfile b/testsuite/dockerdata/Dockerfile
> new file mode 100644
> index 00000000..7372c752
> --- /dev/null
> +++ b/testsuite/dockerdata/Dockerfile
> @@ -0,0 +1,48 @@
> +FROM debian:bullseye-slim as isar-docker

Why not choose kas-isar as parent image and just add the CI specific 
needs? Is there a reason why you want to maintain this a second time?

Best,
Tobias

> +
> +ARG TARGETPLATFORM
> +ARG DEBIAN_FRONTEND=noninteractive
> +
> +RUN apt-get update && \
> +    apt-get install -y locales && \
> +    localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
> +
> +ENV LANG=en_US.utf8
> +ENV LC_ALL=en_US.UTF-8
> +
> +# Isar main dependencies
> +RUN apt-get install -y -f --no-install-recommends \
> +        binfmt-support bzip2 debootstrap dosfstools dpkg-dev gettext-base \
> +        git mtools parted python3 quilt qemu-user-static reprepro sudo \
> +        unzip xz-utils git-buildpackage pristine-tar sbuild schroot zstd \
> +        python3-distutils \
> +        umoci skopeo
> +
> +# Isar testsuite dependencies
> +RUN apt-get install --no-install-recommends -y \
> +        python3-pip && \
> +    pip3 --proxy=$https_proxy install avocado-framework==100.1 && \
> +    rm -rf $(pip3 cache dir) && \
> +    apt-get install -y -f --no-install-recommends \
> +        qemu-system ovmf
> +
> +RUN apt-get clean && \
> +    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
> +
> +COPY contrib/oe-git-proxy /usr/bin/
> +ENV GIT_PROXY_COMMAND="oe-git-proxy" \
> +    NO_PROXY="*"
> +
> +RUN echo "builder ALL=NOPASSWD: ALL" > /etc/sudoers.d/builder-nopasswd && \
> +    chmod 660 /etc/sudoers.d/builder-nopasswd && \
> +    echo "Defaults env_keep += \"ftp_proxy http_proxy https_proxy no_proxy\"" \
> +    > /etc/sudoers.d/env_keep && chmod 660 /etc/sudoers.d/env_keep
> +
> +RUN useradd builder --user-group --create-home --home-dir /builder && \
> +    sbuild-adduser builder >/dev/null 2>/dev/null
> +
> +COPY container-entrypoint /
> +
> +USER builder
> +
> +ENTRYPOINT ["/container-entrypoint"]
> diff --git a/testsuite/dockerdata/container-entrypoint b/testsuite/dockerdata/container-entrypoint
> new file mode 100755
> index 00000000..6af07932
> --- /dev/null
> +++ b/testsuite/dockerdata/container-entrypoint
> @@ -0,0 +1,9 @@
> +#!/bin/bash
> +
> +sudo update-binfmts --enable && [ -f /proc/sys/fs/binfmt_misc/status ]
> +
> +if [ -n "$1" ]; then
> +    exec "$@"
> +else
> +    exec bash
> +fi
> diff --git a/testsuite/dockerdata/contrib/oe-git-proxy b/testsuite/dockerdata/contrib/oe-git-proxy
> new file mode 100755
> index 00000000..aa9b9dc9
> --- /dev/null
> +++ b/testsuite/dockerdata/contrib/oe-git-proxy
> @@ -0,0 +1,187 @@
> +#!/bin/bash
> +
> +# oe-git-proxy is a simple tool to be via GIT_PROXY_COMMAND. It uses socat
> +# to make SOCKS5 or HTTPS proxy connections.
> +# It uses ALL_PROXY or all_proxy or http_proxy to determine the proxy server,
> +# protocol, and port.
> +# It uses NO_PROXY to skip using the proxy for a comma delimited list of
> +# hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24). It
> +# is known to work with both bash and dash shells.
> +#
> +# Example ALL_PROXY values:
> +# ALL_PROXY=socks://socks.example.com:1080
> +# ALL_PROXY=https://proxy.example.com:8080
> +#
> +# Copyright (c) 2013, Intel Corporation.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# AUTHORS
> +# Darren Hart <dvhart@linux.intel.com>
> +
> +# disable pathname expansion, NO_PROXY fields could start with "*" or be it
> +set -f
> +
> +if [ $# -lt 2 -o "$1" = '--help' -o "$1" = '-h' ] ; then
> +    echo 'oe-git-proxy: error: the following arguments are required: host port'
> +    echo 'Usage: oe-git-proxy host port'
> +    echo ''
> +    echo 'OpenEmbedded git-proxy - a simple tool to be used via GIT_PROXY_COMMAND.'
> +    echo 'It uses socat to make SOCKS or HTTPS proxy connections.'
> +    echo 'It uses ALL_PROXY to determine the proxy server, protocol, and port.'
> +    echo 'It uses NO_PROXY to skip using the proxy for a comma delimited list'
> +    echo 'of hosts, host globs (*.example.com), IPs, or CIDR masks (192.168.1.0/24).'
> +    echo 'It is known to work with both bash and dash shells.runs native tools'
> +    echo ''
> +    echo 'arguments:'
> +    echo '  host                proxy host to use'
> +    echo '  port                proxy port to use'
> +    echo ''
> +    echo 'options:'
> +    echo '  -h, --help          show this help message and exit'
> +    echo ''
> +    exit 2
> +fi
> +
> +# Locate the netcat binary
> +if [ -z "$SOCAT" ]; then
> +	SOCAT=$(which socat 2>/dev/null)
> +	if [ $? -ne 0 ]; then
> +		echo "ERROR: socat binary not in PATH" 1>&2
> +		exit 1
> +	fi
> +fi
> +METHOD=""
> +
> +# Test for a valid IPV4 quad with optional bitmask
> +valid_ipv4() {
> +	echo $1 | egrep -q "^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(/(3[0-2]|[1-2]?[0-9]))?$"
> +	return $?
> +}
> +
> +# Convert an IPV4 address into a 32bit integer
> +ipv4_val() {
> +	IP="$1"
> +	SHIFT=24
> +	VAL=0
> +	for B in $( echo "$IP" | tr '.' ' ' ); do
> +		VAL=$(($VAL+$(($B<<$SHIFT))))
> +		SHIFT=$(($SHIFT-8))
> +	done
> +	echo "$VAL"
> +}
> +
> +# Determine if two IPs are equivalent, or if the CIDR contains the IP
> +match_ipv4() {
> +	CIDR=$1
> +	IP=$2
> +
> +	if [ -z "${IP%%$CIDR}" ]; then
> +		return 0
> +	fi
> +
> +	# Determine the mask bitlength
> +	BITS=${CIDR##*/}
> +	[ "$BITS" != "$CIDR" ] || BITS=32
> +	if [ -z "$BITS" ]; then
> +		return 1
> +	fi
> +
> +	IPVAL=$(ipv4_val $IP)
> +	IP2VAL=$(ipv4_val ${CIDR%%/*})
> +
> +	# OR in the unmasked bits
> +	for i in $(seq 0 $((32-$BITS))); do
> +		IP2VAL=$(($IP2VAL|$((1<<$i))))
> +		IPVAL=$(($IPVAL|$((1<<$i))))
> +	done
> +
> +	if [ $IPVAL -eq $IP2VAL ]; then
> +		return 0
> +	fi
> +	return 1
> +}
> +
> +# Test to see if GLOB matches HOST
> +match_host() {
> +	HOST=$1
> +	GLOB=$2
> +
> +	if [ -z "${HOST%%*$GLOB}" ]; then
> +		return 0
> +	fi
> +
> +	# Match by netmask
> +	if valid_ipv4 $GLOB; then
> +		for HOST_IP in $(getent ahostsv4 $HOST | grep ' STREAM ' | cut -d ' ' -f 1) ; do
> +			if valid_ipv4 $HOST_IP; then
> +				match_ipv4 $GLOB $HOST_IP
> +				if [ $? -eq 0 ]; then
> +					return 0
> +				fi
> +			fi
> +		done
> +	fi
> +
> +	return 1
> +}
> +
> +# If no proxy is set or needed, just connect directly
> +METHOD="TCP:$1:$2"
> +
> +[ -z "${ALL_PROXY}" ] && ALL_PROXY=$all_proxy
> +[ -z "${ALL_PROXY}" ] && ALL_PROXY=$http_proxy
> +
> +if [ -z "$ALL_PROXY" ]; then
> +	exec $SOCAT STDIO $METHOD
> +fi
> +
> +# Connect directly to hosts in NO_PROXY
> +for H in $( echo "$NO_PROXY" | tr ',' ' ' ); do
> +	if match_host $1 $H; then
> +		exec $SOCAT STDIO $METHOD
> +	fi
> +done
> +
> +# Proxy is necessary, determine protocol, server, and port
> +# extract protocol
> +PROTO=${ALL_PROXY%://*}
> +# strip protocol:// from string
> +ALL_PROXY=${ALL_PROXY#*://}
> +# extract host & port parts:
> +#   1) drop username/password
> +PROXY=${ALL_PROXY##*@}
> +#   2) remove optional trailing /?
> +PROXY=${PROXY%%/*}
> +#   3) extract optional port
> +PORT=${PROXY##*:}
> +if [ "$PORT" = "$PROXY" ]; then
> +	PORT=""
> +fi
> +#   4) remove port
> +PROXY=${PROXY%%:*}
> +
> +# extract username & password
> +PROXYAUTH="${ALL_PROXY%@*}"
> +[ "$PROXYAUTH" = "$ALL_PROXY" ] && PROXYAUTH=
> +[ -n "${PROXYAUTH}" ] && PROXYAUTH=",proxyauth=${PROXYAUTH}"
> +
> +if [ "$PROTO" = "socks" ] || [ "$PROTO" = "socks4a" ]; then
> +	if [ -z "$PORT" ]; then
> +		PORT="1080"
> +	fi
> +	METHOD="SOCKS4A:$PROXY:$1:$2,socksport=$PORT"
> +elif [ "$PROTO" = "socks4" ]; then
> +	if [ -z "$PORT" ]; then
> +		PORT="1080"
> +	fi
> +	METHOD="SOCKS4:$PROXY:$1:$2,socksport=$PORT"
> +else
> +	# Assume PROXY (http, https, etc)
> +	if [ -z "$PORT" ]; then
> +		PORT="8080"
> +	fi
> +	METHOD="PROXY:$PROXY:$1:$2,proxyport=${PORT}${PROXYAUTH}"
> +fi
> +
> +exec $SOCAT STDIO "$METHOD"

Re: [PATCH 1/3] testsuite: Dockerfile for isar-docker image

[Baurzhan Ismagulov]

Thanks Tobias for the quick review,

On 2023-03-09 07:57, Schaffner, Tobias wrote:
> Why not choose kas-isar as parent image and just add the CI specific 
> needs? Is there a reason why you want to maintain this a second time?

Our preferred way would actually be to fix specific versions of avocado and (if
necessary) qemu in the kas-isar image and continue using it. The new image was
created with the minimal Isar requirements (kas-isar has some 300 MB of
additional packages). We'll check kas-isar as parent and let you know. We'll
probably need some sync on it, like user ID handling, which image version will
be used when, planned updates, etc.

With kind regards,
Baurzhan

Re: [PATCH 1/3] testsuite: Dockerfile for isar-docker image

[Uladzimir Bely]

In the email from Thursday, 9 March 2023 12:05:02 +03 user Baurzhan Ismagulov wrote:
> Thanks Tobias for the quick review,
> 
> On 2023-03-09 07:57, Schaffner, Tobias wrote:
> > Why not choose kas-isar as parent image and just add the CI specific 
> > needs? Is there a reason why you want to maintain this a second time?
> 
> Our preferred way would actually be to fix specific versions of avocado and (if
> necessary) qemu in the kas-isar image and continue using it. The new image was
> created with the minimal Isar requirements (kas-isar has some 300 MB of
> additional packages). We'll check kas-isar as parent and let you know. We'll
> probably need some sync on it, like user ID handling, which image version will
> be used when, planned updates, etc.
> 
> With kind regards,
> Baurzhan
> 
> 

Yes, the only reason was just to have the image size as small as possible, but enough to build Isar. Our own image with avocado and qemu inside got size of 1.07GB. It's a bit less than 1.29GB we get in case of basing on kas-isar (823MB) with additional testsuite stuff.
If size does not matter, it would be even better and easier for us to base on kas-isar, with similar version tagging scheme.

Re: [PATCH 1/3] testsuite: Dockerfile for isar-docker image

[Jan Kiszka]

On 09.03.23 10:05, Baurzhan Ismagulov wrote:
> Thanks Tobias for the quick review,
> 
> On 2023-03-09 07:57, Schaffner, Tobias wrote:
>> Why not choose kas-isar as parent image and just add the CI specific 
>> needs? Is there a reason why you want to maintain this a second time?
> 
> Our preferred way would actually be to fix specific versions of avocado and (if
> necessary) qemu in the kas-isar image and continue using it. The new image was
> created with the minimal Isar requirements (kas-isar has some 300 MB of
> additional packages). We'll check kas-isar as parent and let you know. We'll
> probably need some sync on it, like user ID handling, which image version will
> be used when, planned updates, etc.

Given that most Isar users are also kas-isar users, it would not be an
elegant decision to create and maintain an own base image for Isar CI
purposes only.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 1/3] testsuite: Dockerfile for isar-docker image

[Jan Kiszka]

On 09.03.23 10:14, Uladzimir Bely wrote:
> In the email from Thursday, 9 March 2023 12:05:02 +03 user Baurzhan Ismagulov wrote:
>> Thanks Tobias for the quick review,
>>
>> On 2023-03-09 07:57, Schaffner, Tobias wrote:
>>> Why not choose kas-isar as parent image and just add the CI specific 
>>> needs? Is there a reason why you want to maintain this a second time?
>>
>> Our preferred way would actually be to fix specific versions of avocado and (if
>> necessary) qemu in the kas-isar image and continue using it. The new image was
>> created with the minimal Isar requirements (kas-isar has some 300 MB of
>> additional packages). We'll check kas-isar as parent and let you know. We'll
>> probably need some sync on it, like user ID handling, which image version will
>> be used when, planned updates, etc.
>>
>> With kind regards,
>> Baurzhan
>>wise
>>
> 
> Yes, the only reason was just to have the image size as small as possible, but enough to build Isar. Our own image with avocado and qemu inside got size of 1.07GB. It's a bit less than 1.29GB we get in case of basing on kas-isar (823MB) with additional testsuite stuff.
> If size does not matter, it would be even better and easier for us to base on kas-isar, with similar version tagging scheme.

kas-isar is the minimal image needed for building Isar, that's its
purpose. You can read that up in the image description, and we maintain
that description in lock-step with Isar's documentation (at least that
is the goal).

Again, trying to roll your own for testing purposes only will not be
beneficial for the ecosystem.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 0/3] Switch to own isar-docker image

[Henning Schild]

I will not use some container from ilbers. If avocado turns out to be a
problem, we can switch to pytest or something else that is easy to
install. But please not yet another container to cover up the problems.

Henning

Am Tue,  7 Mar 2023 05:36:46 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In comparison with previously used kas-isar image, switch to own
> isar-docker one, that better fit CI needs:
>  - Avocado testsuite preinstalled;
>  - Qemu-system packages for run tests preinstalled;
>  - Potentially, avocado can be run directly, without ci_build.sh.
> 
> Uladzimir Bely (3):
>   testsuite: Dockerfile for isar-docker image
>   testsuite: Add README for isar-docker
>   gitlab-ci: Switch to own isar-docker image
> 
>  .gitlab-ci.yml                            |   3 +-
>  testsuite/dockerdata/.dockerignore        |   3 +
>  testsuite/dockerdata/Dockerfile           |  48 ++++++
>  testsuite/dockerdata/README.md            |  19 +++
>  testsuite/dockerdata/container-entrypoint |   9 ++
>  testsuite/dockerdata/contrib/oe-git-proxy | 187
> ++++++++++++++++++++++ 6 files changed, 268 insertions(+), 1
> deletion(-) create mode 100644 testsuite/dockerdata/.dockerignore
>  create mode 100644 testsuite/dockerdata/Dockerfile
>  create mode 100644 testsuite/dockerdata/README.md
>  create mode 100755 testsuite/dockerdata/container-entrypoint
>  create mode 100755 testsuite/dockerdata/contrib/oe-git-proxy
> 

Re: [PATCH 0/3] Switch to own isar-docker image

[Jan Kiszka]

On 14.03.23 14:49, Henning Schild wrote:
> I will not use some container from ilbers. If avocado turns out to be a
> problem, we can switch to pytest or something else that is easy to
> install. But please not yet another container to cover up the problems.
> 

To be fair, the test env also needs qemu to run certain tests,
irrespective of the test framework. But that is something kas will not
include into its build-only containers. So, there is extra installation
needed. Or an extra container, but that should be based on kas-isar, not
duplicating its base work.

Jan

> Henning
> 
> Am Tue,  7 Mar 2023 05:36:46 +0100
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> 
>> In comparison with previously used kas-isar image, switch to own
>> isar-docker one, that better fit CI needs:
>>  - Avocado testsuite preinstalled;
>>  - Qemu-system packages for run tests preinstalled;
>>  - Potentially, avocado can be run directly, without ci_build.sh.
>>
>> Uladzimir Bely (3):
>>   testsuite: Dockerfile for isar-docker image
>>   testsuite: Add README for isar-docker
>>   gitlab-ci: Switch to own isar-docker image
>>
>>  .gitlab-ci.yml                            |   3 +-
>>  testsuite/dockerdata/.dockerignore        |   3 +
>>  testsuite/dockerdata/Dockerfile           |  48 ++++++
>>  testsuite/dockerdata/README.md            |  19 +++
>>  testsuite/dockerdata/container-entrypoint |   9 ++
>>  testsuite/dockerdata/contrib/oe-git-proxy | 187
>> ++++++++++++++++++++++ 6 files changed, 268 insertions(+), 1
>> deletion(-) create mode 100644 testsuite/dockerdata/.dockerignore
>>  create mode 100644 testsuite/dockerdata/Dockerfile
>>  create mode 100644 testsuite/dockerdata/README.md
>>  create mode 100755 testsuite/dockerdata/container-entrypoint
>>  create mode 100755 testsuite/dockerdata/contrib/oe-git-proxy
>>
> 

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 0/3] Switch to own isar-docker image

[Henning Schild]

Am Tue, 14 Mar 2023 15:00:53 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 14.03.23 14:49, Henning Schild wrote:
> > I will not use some container from ilbers. If avocado turns out to
> > be a problem, we can switch to pytest or something else that is
> > easy to install. But please not yet another container to cover up
> > the problems. 
> 
> To be fair, the test env also needs qemu to run certain tests,
> irrespective of the test framework. But that is something kas will not
> include into its build-only containers. So, there is extra
> installation needed. Or an extra container, but that should be based
> on kas-isar, not duplicating its base work.

At the moment we use kas-isar and install the additional things in a
controlled manner. I do not see the problem with that, except for maybe
a very neglectable download/install reduction (given what CI does once
it starts rolling). There are far better places for optimization,
saving time/bandwidth/storage.

Of cause kas is not an isar feature and isar can decide on its own
which container or if to container ... but in the end we at Siemens
will often combine with kas and its containers, so we will use that
anyhow ... no matter what the official container will become, kas will
have to work as well.

Henning

> 
> Jan
> 
> > Henning
> > 
> > Am Tue,  7 Mar 2023 05:36:46 +0100
> > schrieb Uladzimir Bely <ubely@ilbers.de>:
> >   
> >> In comparison with previously used kas-isar image, switch to own
> >> isar-docker one, that better fit CI needs:
> >>  - Avocado testsuite preinstalled;
> >>  - Qemu-system packages for run tests preinstalled;
> >>  - Potentially, avocado can be run directly, without ci_build.sh.
> >>
> >> Uladzimir Bely (3):
> >>   testsuite: Dockerfile for isar-docker image
> >>   testsuite: Add README for isar-docker
> >>   gitlab-ci: Switch to own isar-docker image
> >>
> >>  .gitlab-ci.yml                            |   3 +-
> >>  testsuite/dockerdata/.dockerignore        |   3 +
> >>  testsuite/dockerdata/Dockerfile           |  48 ++++++
> >>  testsuite/dockerdata/README.md            |  19 +++
> >>  testsuite/dockerdata/container-entrypoint |   9 ++
> >>  testsuite/dockerdata/contrib/oe-git-proxy | 187
> >> ++++++++++++++++++++++ 6 files changed, 268 insertions(+), 1
> >> deletion(-) create mode 100644 testsuite/dockerdata/.dockerignore
> >>  create mode 100644 testsuite/dockerdata/Dockerfile
> >>  create mode 100644 testsuite/dockerdata/README.md
> >>  create mode 100755 testsuite/dockerdata/container-entrypoint
> >>  create mode 100755 testsuite/dockerdata/contrib/oe-git-proxy
> >>  
> >   
> 

Re: [PATCH 0/3] Switch to own isar-docker image

[Jan Kiszka]

On 14.03.23 15:25, Henning Schild wrote:
> Am Tue, 14 Mar 2023 15:00:53 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 14.03.23 14:49, Henning Schild wrote:
>>> I will not use some container from ilbers. If avocado turns out to
>>> be a problem, we can switch to pytest or something else that is
>>> easy to install. But please not yet another container to cover up
>>> the problems. 
>>
>> To be fair, the test env also needs qemu to run certain tests,
>> irrespective of the test framework. But that is something kas will not
>> include into its build-only containers. So, there is extra
>> installation needed. Or an extra container, but that should be based
>> on kas-isar, not duplicating its base work.
> 
> At the moment we use kas-isar and install the additional things in a
> controlled manner. I do not see the problem with that, except for maybe
> a very neglectable download/install reduction (given what CI does once
> it starts rolling). There are far better places for optimization,
> saving time/bandwidth/storage.
> 
> Of cause kas is not an isar feature and isar can decide on its own
> which container or if to container ... but in the end we at Siemens
> will often combine with kas and its containers, so we will use that
> anyhow ... no matter what the official container will become, kas will
> have to work as well.

Yep, that's exactly why I voted against having an own base container now.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux