linux_compiler_tensorflow_avec_bazel
Différences
Ci-dessous, les différences entre deux révisions de la page.
Les deux révisions précédentesRévision précédenteProchaine révision | Révision précédente | ||
linux_compiler_tensorflow_avec_bazel [2021/12/11 09:30] – [Sans GPU sur Debian] serge | linux_compiler_tensorflow_avec_bazel [2021/12/20 11:13] (Version actuelle) – [La version Intel de Tensorflow] serge | ||
---|---|---|---|
Ligne 1: | Ligne 1: | ||
====== Linux Compiler Tensorflow avec Bazel ====== | ====== Linux Compiler Tensorflow avec Bazel ====== | ||
+ | <WRAP center round box 60% centeralign> | ||
+ | **L' | ||
+ | Voir **[[movenet|]]** | ||
+ | </ | ||
- | =====Ressources===== | + | =====La version Intel de Tensorflow===== |
- | * **[[https:// | + | |
- | * **[[https:// | + | |
- | * **[[https:// | + | |
- | =====Sans GPU sur Debian===== | + | **[[https:// |
- | Désinstaller les packages python protobuf et tensorflow | + | <code bash> |
+ | python3 -m pip install intel-tensorflow | ||
+ | </ | ||
- | Récupération des sources | + | Donne toujours le message AVX2 et FMA. Mais c'est normal.\\ |
- | cd / | + | Bienvenue dans le monde de Tensoflow: c'est un Warning, mais ce n'est pas forcément vrai !\\ |
- | git clone https:// | + | Mon FPS a augmenté de 10 à 20 % :-P |
- | Installation du nécessaire: | + | **Il est inutile |
- | + | =====Ressources===== | |
- | python3 -m pip install pip six numpy wheel setuptools mock future> | + | |
- | python3 -m pip install keras_applications==1.0.6 --no-deps | + | * **[[https:// |
- | python3 -m pip install keras_preprocessing==1.0.5 --no-deps | + | |
- | + | ||
- | Install Bazel, the build tool used to compile TensorFlow. In my case, after downloading bazel-0.26.0-installer-darwin-x86_64.sh: | + | |
- | + | ||
- | chmod +x bazel-0.26.0-installer-darwin-x86_64.sh ./ | + | |
- | bazel version | + | |
- | + | ||
- | + | ||
- | + | ||
- | =====Avec GPU sur Ubuntu===== | + | |
- | ===Contexte=== | + | |
- | | + | |
- | | + | |
- | * venv | + | |
- | + | ||
- | **L' | + | |
- | + | ||
- | ====Cuda==== | + | |
- | * [[https://fr.wikipedia.org/wiki/Compute_Unified_Device_Architecture|Compute Unified Device Architecture]] CUDA (initialement l' | + | |
- | * **[[installation_de_cuda|Installation de CUDA et Compilation de OpenCV avec CUDA]]** | + | |
- | + | ||
- | + | ||
- | ====CudNN==== | + | |
- | * **[[https:// | + | |
- | + | ||
- | sudo apt install libcudnn8 libcudnn8-dev | + | |
- | ====TensorRT==== | + | Sur les internets, ça en cause: |
- | * **[[https://developer.nvidia.com/tensorrt|NVIDIA® TensorRT™]]** is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer | + | * https:// |
+ | | ||
+ | | ||
+ | | ||
+ | * https:// | ||
- | sudo apt install libnvinfer8 libnvinfer8-dev | + | =====Installation de Bazel===== |
- | + | Bazel est le compilateur de Google. | |
- | ====Bazel | + | |
<code bash> | <code bash> | ||
# Bazel | # Bazel | ||
- | |||
sudo apt install apt-transport-https curl gnupg | sudo apt install apt-transport-https curl gnupg | ||
curl -fsSL https:// | curl -fsSL https:// | ||
Ligne 58: | Ligne 37: | ||
echo "deb [arch=amd64] https:// | echo "deb [arch=amd64] https:// | ||
sudo apt update | sudo apt update | ||
+ | sudo apt install bazel-4.2.1 | ||
+ | </ | ||
- | sudo apt install bazel | + | =====Sans GPU sur Debian===== |
- | + | Désinstaller les packages python tensorflow si ils sont installés. | |
- | # Dépendances | + | <code bash> |
- | python3 -m pip install numpy wheel | + | pip3 list |
+ | # Dans la list, chercher ce qu'il faut désinstaller ... | ||
+ | python3 -m pip uninstall tensorflow intel-tensorflow | ||
+ | ... | ||
+ | </ | ||
+ | Installation du nécessaire: | ||
+ | <code bash> | ||
+ | python3 -m pip install numpy wheel protobuf | ||
python3 -m pip install keras_preprocessing --no-deps | python3 -m pip install keras_preprocessing --no-deps | ||
+ | </ | ||
- | # Récupération des sources de tensorflow | + | Récupération des sources de tensorflow: |
- | mkdir src | + | <code bash> |
- | cd src/ | + | cd / |
- | git clone https:// | + | git clone https:// |
- | cd tensorflow | + | |
./configure | ./configure | ||
</ | </ | ||
- | ===Le plus abcons: Configurer la compilation=== | + | Cela génère un fichier .tf_configure.bazelrc dans le dossier |
- | * **[[https:// | + | Répondre entrée ou N partout, nous allons corriger le fichier |
- | + | ||
- | **Très important** Des questions et des réponses: | + | |
- | * **non à ROCm** (non à Rocky) | + | |
- | * non à TensorRT sinon ça plante | + | |
- | * **non** à clang | + | |
- | * **non à download a fresh release of clang** ça plante | + | |
- | * oui à GCC | + | |
- | * --config: -march=native | + | |
- | * Pour 1060 GTX –> 6.1, Pour GTX 850m –> 5.0 | + | |
- | + | ||
- | <WRAP center round important 60%> | + | |
- | ** Comment compiler avec AVX2 et FMA: répondre -march=native** | + | |
- | </ | + | |
<code bash> | <code bash> | ||
- | $ ./configure | + | build --action_env PYTHON_BIN_PATH=" |
- | You have bazel 4.2.1 installed. | + | build --action_env PYTHON_LIB_PATH="/ |
- | Please specify the location of python. [Default is / | + | build --python_path=" |
+ | build:opt --copt=-march=native | ||
+ | build:opt --host_copt=-march=native | ||
+ | test --flaky_test_attempts=3 | ||
+ | test --test_size_filters=small, | ||
+ | test:v1 --test_tag_filters=-benchmark-test, | ||
+ | test:v1 --build_tag_filters=-benchmark-test, | ||
+ | test:v2 --test_tag_filters=-benchmark-test, | ||
+ | test:v2 --build_tag_filters=-benchmark-test, | ||
+ | </ | ||
- | Found possible Python library paths: | ||
- | / | ||
- | / | ||
- | Please input the desired Python library path to use. Default is [/ | ||
- | Do you wish to build TensorFlow with ROCm support? [y/N]: n | ||
- | No ROCm support will be enabled for TensorFlow. | ||
- | Do you wish to build TensorFlow with CUDA support? [y/N]: y | + | ====Compilation de tensorflow==== |
- | CUDA support will be enabled for TensorFlow. | + | <code bash> |
+ | sudo ln -s / | ||
+ | bazel build -j 4 --copt=-mavx2 --copt=-mfma | ||
+ | ./ | ||
+ | </ | ||
- | Do you wish to build TensorFlow with TensorRT support? [y/N]: n | + | ===Consommation mémoire=== |
- | No TensorRT support will be enabled for TensorFlow. | + | Sur un i5 à 8 threads et 8 Go de Ram: -j 2 pour limiter la consommation de mémoire, sinon Swap de 8 Go insuffisante. |
- | Found CUDA 11.5 in: | + | ===Stop et reprise=== |
- | / | + | Bazel reprend là où il s'est arrêté. Pour nettoyer avant de reprendre: bazel clean |
- | / | + | |
- | Found cuDNN 8 in: | + | |
- | / | + | |
- | / | + | |
- | Please specify a list of comma-separated CUDA compute capabilities you want to build with. | + | ===Success=== |
- | You can find the compute capability of your device at: https://developer.nvidia.com/ | + | <code bash> |
- | Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 5.0 | + | INFO: Elapsed time: 19106.040s, Critical Path: 196.86s |
+ | soit 5.3 heures, perso je trouve | ||
+ | INFO: 11460 processes: 1325 internal, 10135 local. | ||
+ | INFO: Build completed successfully, | ||
+ | </ | ||
- | Do you want to use clang as CUDA compiler? [y/N]: n | + | ====Nettoyage pour recommencer==== |
- | nvcc will be used as CUDA compiler. | + | <code bash> |
+ | bazel clean | ||
+ | </ | ||
- | Please specify which gcc should be used by nvcc as the host compiler. [Default is / | + | ====Installation==== |
- | + | <code bash> | |
- | Please specify optimization flags to use during compilation when bazel option " | + | cd /media/data/ |
- | + | source mon_env/ | |
- | Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n | + | python3 |
- | Not configuring the WORKSPACE for Android builds. | + | python3 |
- | + | ||
- | Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" | + | |
- | --config=mkl | + | |
- | --config=mkl_aarch64 # Build with oneDNN and Compute Library for the Arm Architecture (ACL). | + | |
- | --config=monolithic | + | |
- | --config=numa | + | |
- | --config=dynamic_kernels # | + | |
- | --config=v1 | + | |
- | Preconfigured Bazel build configs to DISABLE default on features: | + | |
- | --config=nogcp | + | |
- | --config=nonccl | + | |
- | Configuration finished | + | |
</ | </ | ||
- | ===Reset de la config=== | + | ====Utilisation==== |
- | bazel clean | + | Le message |
- | + | ||
- | + | ||
- | + | ||
- | ====Compilation Création du package pip Installation==== | + | |
- | + | ||
- | Parce que en plus bazel exige **la** bonne version: | + | |
- | sudo apt update && sudo apt install bazel-4.2.1 | + | |
- | + | ||
- | bazel build // | + | |
- | + | ||
- | Après 5 heures de travail | + | |
- | INFO: Build completed successfully, | + | |
- | + | ||
- | + | ||
- | ./ | + | |
- | python3 -m pip install / | + | |
- | + | ||
- | =====Test===== | + | |
- | * **Ne marche pas !** | + | |
{{tag>ia linux python sb tensorflow}} | {{tag>ia linux python sb tensorflow}} |
linux_compiler_tensorflow_avec_bazel.1639215035.txt.gz · Dernière modification : 2021/12/11 09:30 de serge