[QUERY] wic running in buildchroot-target

[QUERY] wic running in buildchroot-target

[chombourger]

[-- Attachment #1.1: Type: text/plain, Size: 2480 bytes --]

Hello all,

I am continuing my journey on the MIPS architecture.
I wanted to add wks files for my MIPS target (Creator Ci40) and 
do_wic_image failed:

INFO: Creating image(s)...

-rw-r--r-- 1 root root 697932185 Jun 18 13:34 
/tmp/development-image-mel-omni-img-creator-ci40.wic/tmp.wic.n3m8x3xo/rootfs_platforma.1.ext4
Unsupported ioctl: cmd=0x0002
Traceback (most recent call last):
  File "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line 
40, in get_block_size
    binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0))
OSError: [Errno 89] Function not implemented

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line 
103, in __init__
    self.block_size = get_block_size(self._f_image)
  File "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line 
42, in get_block_size
    raise IOError("Unable to determine block size")
OSError: Unable to determine block size

wic appears to be running from buildchroot-target and therefore under 
qemu-<arch>-static
evidently mips does not implement the FIGETBSZ ioctl

My options would be:

(1) check if a more recent version of qemu does implement this ioctl
(2) add support for this ioctl in qemu-user
(3) work-around it in wic with something like:

diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
index 080668e..1ce3ace 100644
--- a/scripts/lib/wic/filemap.py
+++ b/scripts/lib/wic/filemap.py
@@ -40,8 +40,13 @@ def get_block_size(file_obj):
 
     # Get the block size of the host file-system for the image file by 
calling
     # the FIGETBSZ ioctl (number 2).
-    binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
-    return struct.unpack('I', binary_data)[0]
+    try:
+        binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
+        return struct.unpack('I', binary_data)[0]
+    except OSError as err:
+        if err.errno == os.errno.ENOSYS:
+            return 4096
+        raise err
 
 class ErrorNotSupp(Exception):
     """

Ok.

But why are we running wic in buildchroot-target. Comments I have seen in 
the code only say that it is a prerequisite but do not say why (I should 
probably check git logs as well)
I wonder if it should run in buildchroot-host context instead (so we still 
use a host environment we control) but avoid the overhead of qemu emulation 
for DISTRO_ARCH != amd64

What are your thoughts?

Cedric


[-- Attachment #1.2: Type: text/html, Size: 3334 bytes --]

Re: [QUERY] wic running in buildchroot-target

[Jan Kiszka]

On 19.06.19 09:45, chombourger@gmail.com wrote:
> Hello all,
> 
> I am continuing my journey on the MIPS architecture.
> I wanted to add wks files for my MIPS target (Creator Ci40) and do_wic_image failed:
> 
> INFO: Creating image(s)...
> 
> -rw-r--r-- 1 root root 697932185 Jun 18 13:34 
> /tmp/development-image-mel-omni-img-creator-ci40.wic/tmp.wic.n3m8x3xo/rootfs_platforma.1.ext4
> Unsupported ioctl: cmd=0x0002
> Traceback (most recent call last):
>    File "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line 40, 
> in get_block_size
>      binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0))
> OSError: [Errno 89] Function not implemented
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>    File "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line 103, 
> in __init__
>      self.block_size = get_block_size(self._f_image)
>    File "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line 42, 
> in get_block_size
>      raise IOError("Unable to determine block size")
> OSError: Unable to determine block size
> 
> wic appears to be running from buildchroot-target and therefore under 
> qemu-<arch>-static
> evidently mips does not implement the FIGETBSZ ioctl
> 
> My options would be:
> 
> (1) check if a more recent version of qemu does implement this ioctl
> (2) add support for this ioctl in qemu-user

Support for that IOCTL seems arch-independent, but maybe the qemu-user build was 
just missing some define:

qemu/linux-user/ioctls.h
      /* emulated ioctl list */
...
#ifdef FIGETBSZ
      IOCTL(FIGETBSZ, IOC_R, MK_PTR(TYPE_LONG))
#endif

But this is better discussed on qemu-devel. You are using qemu from buster 
already, right?

> (3) work-around it in wic with something like:
> 
> diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
> index 080668e..1ce3ace 100644
> --- a/scripts/lib/wic/filemap.py
> +++ b/scripts/lib/wic/filemap.py
> @@ -40,8 +40,13 @@ def get_block_size(file_obj):
>       # Get the block size of the host file-system for the image file by calling
>       # the FIGETBSZ ioctl (number 2).
> -    binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
> -    return struct.unpack('I', binary_data)[0]
> +    try:
> +        binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
> +        return struct.unpack('I', binary_data)[0]
> +    except OSError as err:
> +        if err.errno == os.errno.ENOSYS:
> +            return 4096
> +        raise err
>   class ErrorNotSupp(Exception):
>       """
> 
> Ok.

wic patches should go upstream first. We avoid patching it.

> 
> But why are we running wic in buildchroot-target. Comments I have seen in the 
> code only say that it is a prerequisite but do not say why (I should probably 
> check git logs as well)
> I wonder if it should run in buildchroot-host context instead (so we still use a 
> host environment we control) but avoid the overhead of qemu emulation for 
> DISTRO_ARCH != amd64
> 
> What are your thoughts?

I do not remember all details, there might be technical difficulties, e.g. when 
you need to u-boot tools from a custom build, and that build is currently not 
generating native packages. But the killer argument is: CROSS_COMPILE is only 
optional. The default and production case is native (via qemu-user). Debian is 
not a fully cross-buildable distro, like most others.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [QUERY] wic running in buildchroot-target

[Henning Schild]

Am Wed, 19 Jun 2019 12:28:45 +0200
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 19.06.19 09:45, chombourger@gmail.com wrote:
> > Hello all,
> > 
> > I am continuing my journey on the MIPS architecture.
> > I wanted to add wks files for my MIPS target (Creator Ci40) and

Sweet, we can provide you with a RISC-V board next ;).

> > do_wic_image failed:
> > 
> > INFO: Creating image(s)...
> > 
> > -rw-r--r-- 1 root root 697932185 Jun 18 13:34 
> > /tmp/development-image-mel-omni-img-creator-ci40.wic/tmp.wic.n3m8x3xo/rootfs_platforma.1.ext4
> > Unsupported ioctl: cmd=0x0002
> > Traceback (most recent call last):
> >    File
> > "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line
> > 40, in get_block_size binary_data = fcntl.ioctl(file_obj, 2,
> > struct.pack('I', 0)) OSError: [Errno 89] Function not implemented
> > 
> > During handling of the above exception, another exception occurred:
> > 
> > Traceback (most recent call last):
> >    File
> > "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line
> > 103, in __init__ self.block_size = get_block_size(self._f_image)
> >    File
> > "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line
> > 42, in get_block_size raise IOError("Unable to determine block
> > size") OSError: Unable to determine block size
> > 
> > wic appears to be running from buildchroot-target and therefore
> > under qemu-<arch>-static
> > evidently mips does not implement the FIGETBSZ ioctl
> > 
> > My options would be:
> > 
> > (1) check if a more recent version of qemu does implement this ioctl
> > (2) add support for this ioctl in qemu-user  
> 
> Support for that IOCTL seems arch-independent, but maybe the
> qemu-user build was just missing some define:
> 
> qemu/linux-user/ioctls.h
>       /* emulated ioctl list */
> ...
> #ifdef FIGETBSZ
>       IOCTL(FIGETBSZ, IOC_R, MK_PTR(TYPE_LONG))
> #endif
> 
> But this is better discussed on qemu-devel. You are using qemu from
> buster already, right?
> 
> > (3) work-around it in wic with something like:
> > 
> > diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
> > index 080668e..1ce3ace 100644
> > --- a/scripts/lib/wic/filemap.py
> > +++ b/scripts/lib/wic/filemap.py
> > @@ -40,8 +40,13 @@ def get_block_size(file_obj):
> >       # Get the block size of the host file-system for the image
> > file by calling # the FIGETBSZ ioctl (number 2).
> > -    binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
> > -    return struct.unpack('I', binary_data)[0]
> > +    try:
> > +        binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
> > +        return struct.unpack('I', binary_data)[0]
> > +    except OSError as err:
> > +        if err.errno == os.errno.ENOSYS:
> > +            return 4096
> > +        raise err
> >   class ErrorNotSupp(Exception):
> >       """
> > 
> > Ok.  
> 
> wic patches should go upstream first. We avoid patching it.

In fact we carry a pretty old version of wic (unpatched), and looking
into updating that is somewhere on the Isar TODO list. That might be
fixed in upstream wic, or might be an issue to bring up there to
motivate the patch.
But upstream wic would never run on mips ... so maybe not.

Henning

> > 
> > But why are we running wic in buildchroot-target. Comments I have
> > seen in the code only say that it is a prerequisite but do not say
> > why (I should probably check git logs as well)
> > I wonder if it should run in buildchroot-host context instead (so
> > we still use a host environment we control) but avoid the overhead
> > of qemu emulation for DISTRO_ARCH != amd64
> > 
> > What are your thoughts?  
> 
> I do not remember all details, there might be technical difficulties,
> e.g. when you need to u-boot tools from a custom build, and that
> build is currently not generating native packages. But the killer
> argument is: CROSS_COMPILE is only optional. The default and
> production case is native (via qemu-user). Debian is not a fully
> cross-buildable distro, like most others.
> 
> Jan
> 

Re: [QUERY] wic running in buildchroot-target

[Jan Kiszka]

On 19.06.19 13:10, Henning Schild wrote:
> Am Wed, 19 Jun 2019 12:28:45 +0200
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> On 19.06.19 09:45, chombourger@gmail.com wrote:
>>> Hello all,
>>>
>>> I am continuing my journey on the MIPS architecture.
>>> I wanted to add wks files for my MIPS target (Creator Ci40) and
> 
> Sweet, we can provide you with a RISC-V board next ;).
> 

Smart move: You are trying to avoid that someone drops that board on your desk. ;)

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [QUERY] wic running in buildchroot-target

[chombourger]

Thanks for all the inputs and yes I have been contemplating the idea of getting one of those new little toys (RISC V) but more as a hobby project at this time ;)

Sent from a mobile

> On 19 Jun 2019, at 13:10, Henning Schild <henning.schild@siemens.com> wrote:
> 
> Am Wed, 19 Jun 2019 12:28:45 +0200
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>>> On 19.06.19 09:45, chombourger@gmail.com wrote:
>>> Hello all,
>>> 
>>> I am continuing my journey on the MIPS architecture.
>>> I wanted to add wks files for my MIPS target (Creator Ci40) and
> 
> Sweet, we can provide you with a RISC-V board next ;).
> 
>>> do_wic_image failed:
>>> 
>>> INFO: Creating image(s)...
>>> 
>>> -rw-r--r-- 1 root root 697932185 Jun 18 13:34 
>>> /tmp/development-image-mel-omni-img-creator-ci40.wic/tmp.wic.n3m8x3xo/rootfs_platforma.1.ext4
>>> Unsupported ioctl: cmd=0x0002
>>> Traceback (most recent call last):
>>>   File
>>> "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line
>>> 40, in get_block_size binary_data = fcntl.ioctl(file_obj, 2,
>>> struct.pack('I', 0)) OSError: [Errno 89] Function not implemented
>>> 
>>> During handling of the above exception, another exception occurred:
>>> 
>>> Traceback (most recent call last):
>>>   File
>>> "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line
>>> 103, in __init__ self.block_size = get_block_size(self._f_image)
>>>   File
>>> "/home/chombourger/Projects/isar/scripts/lib/wic/filemap.py", line
>>> 42, in get_block_size raise IOError("Unable to determine block
>>> size") OSError: Unable to determine block size
>>> 
>>> wic appears to be running from buildchroot-target and therefore
>>> under qemu-<arch>-static
>>> evidently mips does not implement the FIGETBSZ ioctl
>>> 
>>> My options would be:
>>> 
>>> (1) check if a more recent version of qemu does implement this ioctl
>>> (2) add support for this ioctl in qemu-user  
>> 
>> Support for that IOCTL seems arch-independent, but maybe the
>> qemu-user build was just missing some define:
>> 
>> qemu/linux-user/ioctls.h
>>      /* emulated ioctl list */
>> ...
>> #ifdef FIGETBSZ
>>      IOCTL(FIGETBSZ, IOC_R, MK_PTR(TYPE_LONG))
>> #endif
>> 
>> But this is better discussed on qemu-devel. You are using qemu from
>> buster already, right?
>> 
>>> (3) work-around it in wic with something like:
>>> 
>>> diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
>>> index 080668e..1ce3ace 100644
>>> --- a/scripts/lib/wic/filemap.py
>>> +++ b/scripts/lib/wic/filemap.py
>>> @@ -40,8 +40,13 @@ def get_block_size(file_obj):
>>>      # Get the block size of the host file-system for the image
>>> file by calling # the FIGETBSZ ioctl (number 2).
>>> -    binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
>>> -    return struct.unpack('I', binary_data)[0]
>>> +    try:
>>> +        binary_data = ioctl(file_obj, 2, struct.pack('I', 0))
>>> +        return struct.unpack('I', binary_data)[0]
>>> +    except OSError as err:
>>> +        if err.errno == os.errno.ENOSYS:
>>> +            return 4096
>>> +        raise err
>>>  class ErrorNotSupp(Exception):
>>>      """
>>> 
>>> Ok.  
>> 
>> wic patches should go upstream first. We avoid patching it.
> 
> In fact we carry a pretty old version of wic (unpatched), and looking
> into updating that is somewhere on the Isar TODO list. That might be
> fixed in upstream wic, or might be an issue to bring up there to
> motivate the patch.
> But upstream wic would never run on mips ... so maybe not.
> 
> Henning
> 
>>> 
>>> But why are we running wic in buildchroot-target. Comments I have
>>> seen in the code only say that it is a prerequisite but do not say
>>> why (I should probably check git logs as well)
>>> I wonder if it should run in buildchroot-host context instead (so
>>> we still use a host environment we control) but avoid the overhead
>>> of qemu emulation for DISTRO_ARCH != amd64
>>> 
>>> What are your thoughts?  
>> 
>> I do not remember all details, there might be technical difficulties,
>> e.g. when you need to u-boot tools from a custom build, and that
>> build is currently not generating native packages. But the killer
>> argument is: CROSS_COMPILE is only optional. The default and
>> production case is native (via qemu-user). Debian is not a fully
>> cross-buildable distro, like most others.
>> 
>> Jan
>> 
>