Attempting to install PVLIB

Been trying over a week to install pvlib, which seems to require a ton of other packages like pandas, numpy, scipy, which in turn need their own compilers, etc… its quite a headache, especially when you consider that nearly every Stackoverflow or equivalent forum expects that you can just type whatever you want into your dockerfile willy-nilly, unlike Toradex’s approach with the configuration panel.

So here is where I stand, I get the following excerpt from a gigantic output:

Questions

  1. Is there a way to stop the system from overwriting the dockerfile?
  2. Is there a way to add multiple ENV variables? something like:

ENV BLAS_SRC=“BLAS (Basic Linear Algebra Subprograms)” \ ENV BLAS=“BLAS (Basic Linear Algebra Subprograms)

  1. Here are the extrapackages I am installing, some of them seemed to have helped because I get “FOUND” instead of “NOT AVAILABLE” but I seem to have a long way to go:

python3-scipy libblas-dev liblapack-dev libatlas-base-dev libblas3 liblapack3 liblapack-doc build-essential iproute2 python-dev python3-pexpect tzdata rename iputils-ping gfortran

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 3.6 MB/s eta 0:00:00
[08-27 15:09:14.845] Collecting h5py
[08-27 15:09:15.409] Downloading h5py-3.7.0.tar.gz (392 kB)
[08-27 15:09:15.514] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 392.4/392.4 kB 861.6 kB/s eta 0:00:00
[08-27 15:09:15.947] Installing build dependencies: started
[08-27 15:10:34.106] Installing build dependencies: still running…
[08-27 15:12:07.646] Installing build dependencies: still running…
[08-27 15:12:15.858] Installing build dependencies: finished with status ‘error’
[08-27 15:12:16.080] e[91m error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [293 lines of output]
Ignoring Cython: markers ‘python_version < “3.8”’ don’t match your environment
Ignoring Cython: markers ‘python_version == “3.8”’ don’t match your environment
Collecting setuptools
Downloading setuptools-65.4.0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 194.8 kB/s eta 0:00:00
Collecting wheel
Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting oldest-supported-numpy
Downloading oldest_supported_numpy-2022.8.16-py3-none-any.whl (3.9 kB)
Collecting pkgconfig
Downloading pkgconfig-1.5.5-py3-none-any.whl (6.7 kB)
Collecting Cython>=0.29.15
Downloading Cython-0.29.32-py2.py3-none-any.whl (986 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 986.3/986.3 kB 273.3 kB/s eta 0:00:00
Collecting numpy==1.19.3
Downloading numpy-1.19.3.zip (7.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 535.5 kB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status ‘done’
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status ‘done’
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): still running…
Preparing metadata (pyproject.toml): finished with status ‘error’
error: subprocess-exited-with-error

    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [254 lines of output]
        Running from numpy source directory.
        setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
          run_build = parse_setuppy_commands()
        Processing numpy/random/_bounded_integers.pxd.in
        Processing numpy/random/_pcg64.pyx
        Processing numpy/random/_philox.pyx
        Processing numpy/random/mtrand.pyx
        Processing numpy/random/_generator.pyx
        Processing numpy/random/_common.pyx
        Processing numpy/random/_mt19937.pyx
        Processing numpy/random/_sfc64.pyx
        Processing numpy/random/_bounded_integers.pyx.in
        Processing numpy/random/bit_generator.pyx
        Cythonizing sources
        blas_opt_info:
        blas_mkl_info:
        customize UnixCCompiler
          libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        blis_info:
          libraries blis not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        openblas_info:
          libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        atlas_3_10_blas_threads_info:
        Setting PTATLAS=ATLAS
          libraries tatlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf/atlas', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        atlas_3_10_blas_info:
          libraries satlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf/atlas', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        atlas_blas_threads_info:
        Setting PTATLAS=ATLAS
          libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf/atlas', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        atlas_blas_info:
        get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'nv', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
        customize Gnu95FCompiler
        Found executable /usr/bin/gfortran
        customize Gnu95FCompiler
        customize Gnu95FCompiler using config
        compiling '_configtest.c':
  
        /* This file is generated from numpy/distutils/system_info.py */
        void ATL_buildinfo(void);
        int main(void) {
          ATL_buildinfo();
          return 0;
        }
  
        C compiler: arm-linux-gnueabihf-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-jS0VHk/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-jS0VHk/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
  
        compile options: '-c'
        arm-linux-gnueabihf-gcc: _configtest.c
        arm-linux-gnueabihf-gcc -pthread _configtest.o -lf77blas -lcblas -latlas -o _configtest
          FOUND:
            language = c
            define_macros = [('HAVE_CBLAS', None), ('NO_ATLAS_INFO', -1)]
            libraries = ['f77blas', 'cblas', 'atlas', 'f77blas', 'cblas']
            library_dirs = ['/usr/lib/arm-linux-gnueabihf']
  
          FOUND:
            language = c
            define_macros = [('HAVE_CBLAS', None), ('NO_ATLAS_INFO', -1)]
            libraries = ['f77blas', 'cblas', 'atlas', 'f77blas', 'cblas']
            library_dirs = ['/usr/lib/arm-linux-gnueabihf']
  
        non-existing path in 'numpy/distutils': 'site.cfg'
        lapack_opt_info:
        lapack_mkl_info:
          libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        openblas_lapack_info:
          libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        openblas_clapack_info:
          libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        flame_info:
          libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
          NOT AVAILABLE
  
        atlas_3_10_threads_info:
        Setting PTATLAS=ATLAS
          libraries lapack_atlas not found in /usr/local/lib
          libraries tatlas,tatlas not found in /usr/local/lib
          libraries lapack_atlas not found in /usr/lib
          libraries tatlas,tatlas not found in /usr/lib
          libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries tatlas,tatlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries tatlas,tatlas not found in /usr/lib/arm-linux-gnueabihf
        <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
          NOT AVAILABLE
  
        atlas_3_10_info:
          libraries lapack_atlas not found in /usr/local/lib
          libraries satlas,satlas not found in /usr/local/lib
          libraries lapack_atlas not found in /usr/lib
          libraries satlas,satlas not found in /usr/lib
          libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries satlas,satlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries satlas,satlas not found in /usr/lib/arm-linux-gnueabihf
        <class 'numpy.distutils.system_info.atlas_3_10_info'>
          NOT AVAILABLE
  
        atlas_threads_info:
        Setting PTATLAS=ATLAS
          libraries lapack_atlas not found in /usr/local/lib
          libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
          libraries lapack_atlas not found in /usr/lib
          libraries ptf77blas,ptcblas,atlas not found in /usr/lib
          libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries ptf77blas,ptcblas,atlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries ptf77blas,ptcblas,atlas not found in /usr/lib/arm-linux-gnueabihf
        <class 'numpy.distutils.system_info.atlas_threads_info'>
          NOT AVAILABLE
  
        atlas_info:
          libraries lapack_atlas not found in /usr/local/lib
          libraries f77blas,cblas,atlas not found in /usr/local/lib
          libraries lapack_atlas not found in /usr/lib
          libraries f77blas,cblas,atlas not found in /usr/lib
          libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf/atlas
          libraries f77blas,cblas,atlas not found in /usr/lib/arm-linux-gnueabihf/atlas
        <class 'numpy.distutils.system_info.atlas_info'>
        get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'nv', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
        customize Gnu95FCompiler
        customize Gnu95FCompiler
        customize Gnu95FCompiler using config
        compiling '_configtest.c':
  
        /* This file is generated from numpy/distutils/system_info.py */
        void ATL_buildinfo(void);
        int main(void) {
          ATL_buildinfo();
          return 0;
        }
  
        C compiler: arm-linux-gnueabihf-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-jS0VHk/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-jS0VHk/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
  
        compile options: '-c'
        arm-linux-gnueabihf-gcc: _configtest.c
        arm-linux-gnueabihf-gcc -pthread _configtest.o -lf77blas -lcblas -latlas -o _configtest
          FOUND:
            language = f77
            libraries = ['lapack', 'f77blas', 'cblas', 'atlas', 'f77blas', 'cblas']
            library_dirs = ['/usr/lib/arm-linux-gnueabihf']
            define_macros = [('NO_ATLAS_INFO', -1)]
  
          FOUND:
            language = f77
            libraries = ['lapack', 'f77blas', 'cblas', 'atlas', 'f77blas', 'cblas']
            library_dirs = ['/usr/lib/arm-linux-gnueabihf']
            define_macros = [('NO_ATLAS_INFO', -1)]
  
        /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
          warnings.warn(msg)
        running dist_info
        running build_src
        build_src
        building py_modules sources
        creating build
        creating build/src.linux-armv7l-3.9
        creating build/src.linux-armv7l-3.9/numpy
        creating build/src.linux-armv7l-3.9/numpy/distutils
        building library "npymath" sources
        creating build/src.linux-armv7l-3.9/numpy/core
        creating build/src.linux-armv7l-3.9/numpy/core/src
        creating build/src.linux-armv7l-3.9/numpy/core/src/npymath
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npymath/npy_math_internal.h
          adding 'build/src.linux-armv7l-3.9/numpy/core/src/npymath' to include_dirs.
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npymath/ieee754.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npymath/npy_math_complex.c
        None - nothing done with h_files = ['build/src.linux-armv7l-3.9/numpy/core/src/npymath/npy_math_internal.h']
        building library "npysort" sources
        creating build/src.linux-armv7l-3.9/numpy/core/src/common
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/common/npy_sort.h
          adding 'build/src.linux-armv7l-3.9/numpy/core/src/common' to include_dirs.
        creating build/src.linux-armv7l-3.9/numpy/core/src/npysort
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/quicksort.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/mergesort.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/timsort.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/heapsort.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/radixsort.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/common/npy_partition.h
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/selection.c
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/common/npy_binsearch.h
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/npysort/binsearch.c
        None - nothing done with h_files = ['build/src.linux-armv7l-3.9/numpy/core/src/common/npy_sort.h', 'build/src.linux-armv7l-3.9/numpy/core/src/common/npy_partition.h', 'build/src.linux-armv7l-3.9/numpy/core/src/common/npy_binsearch.h']
        building library "npyrandom" sources
        building extension "numpy.core._multiarray_tests" sources
        creating build/src.linux-armv7l-3.9/numpy/core/src/multiarray
        conv_template:> build/src.linux-armv7l-3.9/numpy/core/src/multiarray/_multiarray_tests.c
        building extension "numpy.core._multiarray_umath" sources
        Traceback (most recent call last):
          File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
            main()
          File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
            return hook(metadata_directory, config_settings)
          File "/tmp/pip-build-env-ag4p0hdp/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
            self.run_setup()
          File "/tmp/pip-build-env-ag4p0hdp/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/tmp/pip-build-env-ag4p0hdp/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
            exec(compile(code, __file__, 'exec'), locals())
          File "setup.py", line 508, in <module>
            setup_package()
          File "setup.py", line 500, in setup_package
            setup(**metadata)
          File "/tmp/pip-install-xg_3w20w/numpy_46ca8b7ef3c74b5ba5ec053c51ebbcba/numpy/distutils/core.py", line 169, in setup
            return old_setup(**new_attr)
          File "/tmp/pip-build-env-ag4p0hdp/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
            return distutils.core.setup(**attrs)
          File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
            dist.run_commands()
          File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
            self.run_command(cmd)
          File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
            cmd_obj.run()
          File "/tmp/pip-build-env-ag4p0hdp/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 31, in run
            egg_info.run()
          File "/tmp/pip-install-xg_3w20w/numpy_46ca8b7ef3c74b5ba5ec053c51ebbcba/numpy/distutils/command/egg_info.py", line 24, in run
            self.run_command("build_src")
          File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
            self.distribution.run_command(command)
          File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
            cmd_obj.run()
          File "/tmp/pip-install-xg_3w20w/numpy_46ca8b7ef3c74b5ba5ec053c51ebbcba/numpy/distutils/command/build_src.py", line 144, in run
            self.build_sources()
          File "/tmp/pip-install-xg_3w20w/numpy_46ca8b7ef3c74b5ba5ec053c51ebbcba/numpy/distutils/command/build_src.py", line 161, in build_sources
            self.build_extension_sources(ext)
          File "/tmp/pip-install-xg_3w20w/numpy_46ca8b7ef3c74b5ba5ec053c51ebbcba/numpy/distutils/command/build_src.py", line 318, in build_extension_sources
            sources = self.generate_sources(sources, ext)
          File "/tmp/pip-install-xg_3w20w/numpy_46ca8b7ef3c74b5ba5ec053c51ebbcba/numpy/distutils/command/build_src.py", line 378, in generate_sources
            source = func(extension, build_dir)
          File "numpy/core/setup.py", line 436, in generate_config_h
            moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
          File "numpy/core/setup.py", line 47, in check_types
            out = check_types(*a, **kw)
          File "numpy/core/setup.py", line 292, in check_types
            raise SystemError(
        SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
        [end of output]
  
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed
  
  × Encountered error while generating package metadata.
  ╰─> See above for output.
  
  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
e[0m
[08-27 15:12:16.086] e[91merror: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
e[0m
[08-27 15:12:32.572] Local docker exception.
[08-27 15:12:32.573] Error (530) - Docker exception: The command '/bin/sh -c dos2unix /requirements.txt && dos2unix /setup.sh && dos2unix /cleanup.sh && chmod a+x /setup.sh && ch

Greetings @leighjboyd,

It’s a known issue that pip is not very reliable when targeting arm32. This is because many packages do not provide a pre-built arm32 binary. This is why things like trying to install numpy require compilers and other libraries and such. Since pip can’t locate a pre-built binary it instead fetches the source of each package and attempts to build from scratch. This causes the dependency hell you’re experiencing at the moment.

A simple workaround would be to use 3rd party index that provides pre-built binaries for the packages you’re seeking. For arm32 a good one is: https://www.piwheels.org/

This index seems to have pre-built binaries for pvlib as well as it’s dependencies. Meaning you won’t need to worry about installing dev libraries and other compilers.

To use this index just have this in your requirements.txt file:

-i https://www.piwheels.org/simple 
pvlib

Best Regards,
Jeremias

Okay, I worked around by doing my own math with math.h only.
it wasn’t that complicated.
Thanks