Différences

Ci-dessous, les différences entre deux révisions de la page.

--- linux_compiler_tensorflow_avec_bazel [2021/12/10 19:38] – [Bazel et Tensorflow] serge
+++ linux_compiler_tensorflow_avec_bazel [2021/12/20 11:13] (Version actuelle) – [La version Intel de Tensorflow] serge
@@ Ligne 1: / Ligne 1: @@
 ====== Linux Compiler Tensorflow avec Bazel ======
-<WRAP center round alert 60%>
+<WRAP center round box 60% centeralign>
-**Aucun tuto correct sur tous les internets, la doc de Google est maximaliste mais mal structurée, aucun exemple, comment définir des config ? Mystère**\\
+**L'objectif est de compiler Tensorflow avec les options AVX2 et FMA et sans GPU**\\
-**Si j'y arrive ce tuto va valoir de l'or !**
+Voir **[[movenet|]]**
 </WRAP>
-Les téléchargements sont très gros: ce sont **des paquets** de **500 Mo** !
+=====La version Intel de Tensorflow=====
+**[[https://pypi.org/project/intel-tensorflow/|intel-tensorflow]]**
+<code bash>
+python3 -m pip install intel-tensorflow
+</code>
+Donne toujours le message AVX2 et FMA. Mais c'est normal.\\
+Bienvenue dans le monde de Tensoflow: c'est un Warning, mais ce n'est pas forcément vrai !\\
+Mon FPS a augmenté de 10 à 20 % :-P
+**Il est inutile de compiler** 8-)
 =====Ressources=====
   * **[[https://www.tensorflow.org/install/source|Compiler à partir de la source]]**
-  * **[[https://docs.bazel.build/versions/main/install-ubuntu.html|Installing Bazel on Ubuntu]]**
+  * **[[https://www.intel.com/content/www/us/en/developer/articles/guide/optimization-for-tensorflow-installation-guide.html|Build TensorFlow from Source with Intel oneAPI oneDNN library]]**
-===Contexte===
+Sur les internets, ça en cause:
-  * Ubuntu Mate 20.04
+  * https://docs.bazel.build/versions/main/install-ubuntu.html
-  * python 3.8
+  * https://technofob.com/2019/06/14/how-to-compile-tensorflow-2-0-with-avx2-fma-instructions-on-mac
-  * venv
+  * https://github.com/tensorflow/tensorflow/issues/7660
+  * https://ai-pool.com/d/proper-way-to-compile-tensorflow-with-sse4-2-and-avx
+  * https://github.com/GTGraphics3/tensorflow-1.4.2-AVX-MKL
-**L'objectif est de compiler Tensorflow avec les options AVX2 et FMA !**
+=====Installation de Bazel=====
+Bazel est le compilateur de Google.
-====Cuda====
-  * [[https://fr.wikipedia.org/wiki/Compute_Unified_Device_Architecture|Compute Unified Device Architecture]] CUDA (initialement l'acronyme de Compute Unified Device Architecture) est une technologie de GPGPU (General-Purpose Computing on Graphics Processing Units), c'est-à-dire utilisant un processeur graphique (GPU) pour exécuter des calculs généraux à la place du processeur central (CPU).
-  * **[[installation_de_cuda|Installation de CUDA et Compilation de OpenCV avec CUDA]]**
-====CudNN====
-  * **[[https://developer.nvidia.com/cudnn|cuDNN]]** is a framework built on the top of CUDA and developed by NVIDIA for deep learning primitives. It stands for CUDA Deep Neural Network. It serves as a building block for deep learning and machine learning frameworks.
-  sudo apt install libcudnn8 libcudnn8-dev
-====TensorRT====
-  * **[[https://developer.nvidia.com/tensorrt|NVIDIA® TensorRT™]]** is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for deep learning inference applications.
-  sudo apt install libnvinfer8 libnvinfer8-dev
-====Bazel et Tensorflow====
 <code bash>
 # Bazel
 sudo apt install apt-transport-https curl gnupg
 curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
@@ Ligne 42: / Ligne 37: @@
 echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
 sudo apt update
+sudo apt install bazel-4.2.1
+</code>
-sudo apt install bazel
+=====Sans GPU sur Debian=====
+Désinstaller les packages python tensorflow si ils sont installés.
-# Dépendances
+<code bash>
-python3 -m pip install numpy wheel
+pip3 list
+# Dans la list, chercher ce qu'il faut désinstaller ...
+python3 -m pip uninstall tensorflow intel-tensorflow
+...
+</code>
+Installation du nécessaire:
+<code bash>
+python3 -m pip install numpy wheel protobuf
 python3 -m pip install keras_preprocessing --no-deps
+</code>
-# Récupération des sources de tensorflow
+Récupération des sources de tensorflow:
-mkdir src
+<code bash>
-cd src/
+cd /votre/dossier/src
-git clone https://github.com/tensorflow/tensorflow.git
+git clone https://github.com/tensorflow/tensorflow
-cd tensorflow
 ./configure
 </code>
-===Le plus abcons: Configurer la compilation===
+Cela génère un fichier .tf_configure.bazelrc dans le dossier tensorflow (c'est un fichier caché: Ctrl+h).\\
-  * **[[https://www.tensorflow.org/install/source#configure_the_build|Configurer la compilation]]**
+Répondre entrée ou N partout, nous allons corriger le fichier à la main:
-**Très important** Des questions et des réponses:
-  * **non à ROCm** (non à Rocky)
-  * **non** à clang
-  * **non à download a fresh release of clang** ça plante
-  * oui à GCC
-  * --config: -march=native
-  * Pour 1060 GTX –> 6.1, Pour GTX 850m –> 5.0
-<WRAP center round important 60%>
-** Comment compiler avec AVX2 et FMA: répondre -march=native**
-</WRAP>
 <code bash>
-$ ./configure
+build --action_env PYTHON_BIN_PATH="/usr/bin/python3"
-You have bazel 4.2.1 installed.
+build --action_env PYTHON_LIB_PATH="/usr/lib/python3/dist-packages"
-Please specify the location of python. [Default is /usr/bin/python3]:
+build --python_path="/usr/bin/python3"
+build:opt --copt=-march=native
+build:opt --host_copt=-march=native
+test --flaky_test_attempts=3
+test --test_size_filters=small,medium
+test:v1 --test_tag_filters=-benchmark-test,-no_oss,-gpu,-oss_serial
+test:v1 --build_tag_filters=-benchmark-test,-no_oss,-gpu
+test:v2 --test_tag_filters=-benchmark-test,-no_oss,-gpu,-oss_serial,-v1only
+test:v2 --build_tag_filters=-benchmark-test,-no_oss,-gpu,-v1only
+</code>
-Found possible Python library paths:
-  /usr/lib/python3/dist-packages
-  /usr/local/lib/python3.8/dist-packages
-Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]
-Do you wish to build TensorFlow with ROCm support? [y/N]: n
-No ROCm support will be enabled for TensorFlow.
-Do you wish to build TensorFlow with CUDA support? [y/N]: y
+====Compilation de tensorflow====
-CUDA support will be enabled for TensorFlow.
+<code bash>
+sudo ln -s /usr/bin/python3 /usr/bin/python
+bazel build -j 4  --copt=-mavx2 --copt=-mfma  //tensorflow/tools/pip_package:build_pip_package
+./bazel-bin/tensorflow/tools/pip_package/build_pip_package /media/data/tensorflow_pkg
+</code>
-Do you wish to build TensorFlow with TensorRT support? [y/N]: n
+===Consommation mémoire===
-No TensorRT support will be enabled for TensorFlow.
+Sur un i5 à 8 threads et 8 Go de Ram: -j 2 pour limiter la consommation de mémoire, sinon Swap de 8 Go insuffisante.
-Found CUDA 11.5 in:
+===Stop et reprise===
-    /usr/local/cuda-11.5/targets/x86_64-linux/lib
+Bazel reprend là où il s'est arrêté.  Pour nettoyer avant de reprendre: bazel clean
-    /usr/local/cuda-11.5/targets/x86_64-linux/include
-Found cuDNN 8 in:
-    /usr/lib/x86_64-linux-gnu
-    /usr/include
-Please specify a list of comma-separated CUDA compute capabilities you want to build with.
+===Success===
-You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
+<code bash>
-Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 5.0
+INFO: Elapsed time: 19106.040s, Critical Path: 196.86s
+soit 5.3 heures, perso je trouve 7 heures!
+INFO: 11460 processes: 1325 internal, 10135 local.
+INFO: Build completed successfully, 11460 total actions
+</code>
-Do you want to use clang as CUDA compiler? [y/N]: n
+====Nettoyage pour recommencer====
-nvcc will be used as CUDA compiler.
+<code bash>
+bazel clean
+</code>
-Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
+====Installation====
+<code bash>
-Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: -march=native
+cd /media/data/3D/projets/grande_echelle_move
+source mon_env/bin/activate
-Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
+python3 -m pip uninstall tensorflow
-Not configuring the WORKSPACE for Android builds.
+python3 -m pip install /media/data/src/tensorflow-2.8.0-cp38-cp38-linux_x86_64.whl
-Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
-	--config=mkl         	# Build with MKL support.
-	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
-	--config=monolithic  	# Config for mostly static monolithic build.
-	--config=numa        	# Build with NUMA support.
-	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
-	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
-Preconfigured Bazel build configs to DISABLE default on features:
-	--config=nogcp       	# Disable GCP support.
-	--config=nonccl      	# Disable NVIDIA NCCL support.
-Configuration finished
 </code>
-===Reset de la config===
+====Utilisation====
-  bazel clean
+Le message d'info AVX2 et FMA est toujours là !
-====Compilation Création du package pip Installation====
-Parce que en plus bazel exige **la** bonne version:
-  sudo apt update && sudo apt install bazel-4.2.1
-  bazel build //tensorflow/tools/pip_package:build_pip_package
-INFO: Build completed successfully, 32129 total actions
-  ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
-  python3 -m pip install /media/data/src/tensorflow-2.8.0-cp38-cp38-linux_x86_64.whl
 {{tag>ia linux python sb tensorflow}}