Hello. When I started learning Python, installed Jupyter Notebook for the first time, then I tried to transfer the application I created to it to the company, I often encountered various problems. That Cyrillic in the user name interferes, then the settings are not transferred, then something else. I mostly overcame all these problems on my own, using Google and spending a lot of time on solving them.
As my experience grew, I learned how to create a folder in which virtual Python is transferred from one computer to another, Jupyter and Matplotlib settings, portable programs (ffmpeg, etc.) and fonts. I could write a program at home, copy the entire folder to the computer of the enterprise, and be sure that nothing will be lost or broken on level ground. Then I thought that this folder could be given to a newbie in Python, and he would get a fully configured and portable environment.
Table of contents
In recent years, Python has become a popular programming language. It is often used for writing mathematical calculations, analyzing big data, machine learning and building neural networks. After the appearance of asinc
and await
constructions, writing fast web frameworks became possible. Python performance gradually increases from release to release, and using Cython or Numba can make an application even faster than in other programming languages. For example, the speed of the web framework Vibora (en) is comparable to the speed of the solutions on the Go (en) . In 2018, Python officially became the language of study in schools and universities of France (en) and Kazakhstan (en) . In Russia, at least some of the departments have switched to Python, for example, the department RK-6 (ru) at MSTU. N.E. Bauman.
When starting to learn Python, new users sometimes have difficulty installing the necessary libraries and setting up the programming environment. If the Windows username contains non-Latin characters, some of the libraries might not install or run. Novice users may have problems configuring the Jupyter Notebook on their local computer. If it is installed on the C:\
drive, how to open the file on the D:\
? When I took the first steps in Python, I also had to overcome these difficulties.
Finally, if all the problems are behind, it can be difficult to transfer the application to another user. I came across a situation where the virtual Python environment I created refused to work on another computer. In addition, Jupyter Notebook and Matplotlib store their settings in the user's folder, which complicates the transfer of applications that use specific settings.
The solution to the problems described above will be to create a fully portable assembly of Jupyter Notebook and / or Jupyter Lab on Windows. It keeps the Python interpreter, its libraries and settings, the settings of all the necessary third-party libraries, including Matplotlib and Jupyter, is not tied to the user name and will not swear if you run it on another computer. We can archive such an assembly into an archive, or write a script or program that will create the same assembly on an absolute beginner's computer. For more advanced users, a portable build may be useful in that it allows you to store the Python environment and settings in libraries in different places. You can place the settings folder in a special place that is synchronized with the cloud storage: Dropbox, Mail.ru * cloud, Yandex or Google. Due to this, all computers will automatically have a locally working environment with the same settings.
* Yes, the one whose client does not connect to Linux anymore (ru) . If they remove the same for Windows, I will have to look for a replacement. 1 TB on the road for free is not lying.
For simplicity, I decided to describe the creation of a portable assembly under Windows. But this instruction with minimal changes is suitable for creating an assembly on Linux and Mac OS. The article is primarily intended for beginners, so I tried to describe as much as possible and easier to read.
The article consists of two parts. In the first part we will create a portable assembly, in the second we will deal with the settings for Jupyter Notebook, Jupyter Lab, IPython and Matplotlib.
Create a folder C:\Dev
. It will install Minconda and the portable assembly Jupyter *.
* Here and below Jupyter = Jupyter Notebook + Juputer Lab.
Download the Miniconda installer from https://conda.io/miniconda (en) . Choose Python 3 for Windows 64 bits or 32 bits depending on the bitness of your operating system. Install Miniconda in the folder C:\Dev\Miniconda3
.
Create the following directory structure for the portable Jupyter build:
C:\ Dev\ Jupyter\ dist\ apps\ conf\ backup\ ipython\ jupyter\ matplotlib\ fonts\ projects\
Create a virtual Python environment with conda
*:
C:\Dev\Miniconda3\Scripts\conda.exe create -p C:\Dev\Jupyter\dist\pyenv3.7-win64 --copy --yes python=3 conda
* You can use the conda-forge
channel to install more recent libraries by adding the argument -c conda-forge
:
C:\Dev\Miniconda3\Scripts\conda.exe create -p C:\Dev\Jupyter\dist\pyenv3.7-win64 --copy --yes -c conda-forge python=3 conda
Activate the environment and install Python packages using pip
*:
C:\Dev\Jupyter\dist\pyenv3.7-win64\Scripts\activate pip --no-cache-dir install numpy scipy matplotlib jupyter jupyterlab
Note: if you need to install Numpy and Scipy, which use Intel's MKL library to speed up calculations, use (en) intel-numpy
instead of numpy
and intel-scipy
instead of scipy
(installed only in Python 3.6!):
pip --no-cache-dir install intel-numpy intel-scipy matplotlib jupyter jupyterlab
After installation, run:
conda.bat deactivate
* If errors occur during installation, try this:
C:\Dev\Jupyter\dist\pyenv3.7-win64\Scirpts\activate conda config --add channels conda-forge conda install numpy scipy matplotlib jupyter jupyterlab
and after installation
conda.bat deactivate
In the folder C:\Dev\Jupyter\dist
create the file setenv.bat
, which will control where Jupyter and Matplotlib will store their settings:
@echo off set conf_path=%~dp0\conf set JUPYTER_CONFIG_DIR=%conf_path%\jupyter set JUPYTER_DATA_DIR=%conf_path%\jupyter\data set JUPYTER_RUNTIME_DIR=%conf_path%\jupyter\data\runtime set IPYTHONDIR=%conf_path%\ipython set MPLCONFIGDIR=%conf_path%\matplotlib REM Matplotlib search FFMPEG in PATH variable only! set PATH=%~dp0\apps\ffmpeg\bin;%PATH%
In the C:\Dev\Jupyter\dist
folder, create a file run_jupyter_notebook.bat
to launch Jupyter Notebook with the specified parameters:
@echo off call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter-notebook.exe --notebook-dir=%1
Similarly, in the folder C:\Dev\Jupyter\dist
create the file run_jupyter_lab.bat
to launch Jupyter Lab with the specified parameters:
@echo off call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter-lab.exe --notebook-dir=%1
In the C:\Dev\Jupyter\dist
folder, create the file enable_extension.bat
, which activates the specified extension in Jupyter Notebook:
@echo off REM Enable extension in Jupyter Notebook. REM Example: REM enable_extension.bat widgetsnbextension call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter-nbextension.exe enable %1
Assume that the work files are in the folder D:\my-projects
. In this folder, create shortcuts to the files run_jupyter_notebook.bat
and run_jupyter_lab.bat
. After creating each of the shortcuts, go to its properties and clear the line "Working folder". If you do not clear - Jupyter will not see the folder you need!
Portable assembly Jupyter created and ready for configuration and operation. To get started, just click on the created shortcuts. If you decide not to delete the installed Miniconda, you can reduce the size of the C:\Dev\Miniconda3
following command:
C:\Dev\Miniconda3\Scripts\conda.exe clean --all
After executing this command, go to the C:\Dev\Miniconda3\pkgs
and clear the contents of the .trash
folder. Only then will we really reduce the size of the Miniconda3
folder.
Let's create in the root of the drive C:\
Dev
folder. In this folder, I add all the programs and development tools that for some reason prefer to be installed not in C:\Program Files
. For example, there I install Ruby, Go, Python, Jupyter, Msys, SQLite Studio, etc.
First we need to install Python. Python has two branches: Python 2 and Python 3. Python 2 is supported (en) until 2020, so we will install only Python 3.
To install Python 3, they usually refer to the official site python.org (en) , from where they download it and install it. However, we want to get a portable build, so we’ll do it differently: we download and install Miniconda.
What is Miniconda? In fact, it is Python with the conda
package manager preinstalled and configured. The conda
console program will allow us to create a folder in which there will be a Python version we need no matter what version of Python comes with as part of Miniconda. Also with the help of conda
in this folder you can install almost all known libraries for Python: Numpy, Scipy, Matplotlib, Sympy, etc. The folder in which Python and its libraries are installed is called a virtual environment. Python libraries come in the form of special archives, called packages.
The conda has distinctive features that make it convenient for both novice and experienced users:
* It should be noted that the situation with installing packages in Python is improving from year to year. A few years ago I was unable to install Numpy via pip
(an error was issued), and I used a conda
. In 2018, I tried the latest version of pip
, and I downloaded a file with the .whl
extension (the so-called “wheel”) with the already compiled Numpy, and everything was fine.
So, we need to download and install Miniconda. To do this, go to https://conda.io/miniconda (en) and select the 64-bit version for Windows in Python 3. If you have a 32-bit computer, you should download the 32-bit version.
Miniconda is installed in the same way as a regular Windows application:
Run the installer, click Next.
We agree with the license agreement I Agree
I prefer the installation for all users, because it will give me the opportunity to specify the path to install. Select the item "All users":
Adjust the path for installation on C:\Dev\Miniconda3
:
Here I put both checkboxes. The checkbox “Add Anaconda to the system PATH environment variable” will make the conda
command available in the terminal from any directory. If you do not check this box, the only thing that will change is that in the terminal instead of conda
you will need to type the full path to conda.exe
. I do not install Anaconda, because it puts a lot of unnecessary things to me, so I ignore the undesirability of setting this flag. If you check this box and change your mind after installation, you can simply remove the conda
from the system variables. It's simple. But if you do not know, you can google or ask. Contact at the end of the article.
I also check the box “Register Anaconda as the system Python 3.7”. If any program suddenly needs Python, it will use Python installed with Miniconda. Also this flag will make the python
command available in the terminal from any folder. This checkbox is recommended if you have not installed Python before. If some Python is already installed, I would not advise checking this box now, but adjust the system variables if necessary.
After that, click Install and the installation process will start:
During installation, you can click Show details. Thereby you will see more information about what exactly happens during installation. But this is optional.
When the installation is complete, the phrase “Completed” will appear, and the Next button will become available. Click Next
In the last window, we are invited to learn about Anaconda Cloud (this is the first box) and how to start working with Anaconda (second box). I don't need any of this, so I clear all the checkboxes and click Finish. Miniconda installation is complete.
After installing Miniconda in the C:\Dev
folder, we will see a new Miniconda
folder weighing approximately 340 MB. Yes, it is a lot, and she will still swell. Later I will show how to reduce its volume quickly and safely.
Go to the folder Miniconda
. Scrolling the file list a bit, we’ll see python.exe
. The same Python 3.7, which was installed in my case (in the screenshot of Directory Opus).
If you double-click on python.exe
, the console window will start, in which you can enter Python commands.
You can test after >>>
enter:
import antigravity
and press Enter. A default browser will open with a Python comic on xkcd .
In the folder C:\Dev\Miniconda\Scripts
we will find conda.exe
. This is the console command with which we will create a virtual Python environment.
Now we are ready to start creating a portable assembly Jupyter Notebook. First, create the following directory structure:
C:\ Dev\ Jupyter\ dist\ apps\ conf\ backup\ ipython\ jupyter\ matplotlib\ fonts\ projects\
In the Dev
folder, create a folder Jupyter
. In turn, in the folder Jupyter
create folders dist
and projects
. In the dist
folder there will be a virtual Python environment with all the necessary libraries, configuration files, additional programs, fonts - all that is necessary for our development in Python in the Jupyter Notebook or Jupyter Lab environment. The projects
folder is the default location for projects. I myself usually do not use this folder, and it remains empty. But if I need to transfer the program to another user along with the configured Jupyter, I will put my program in this projects
folder, make an archive of the entire Jupyter
folder and send the archive to the user.
The apps
folder contains helper programs. For example, I often put there a portable version of FFMPEG that Matplotlib needs to create animations.
The conf
folder contains settings for various libraries. In our case for IPython, Jupyter and Matplotlib.
In the folder conf\backup
I put copies of my settings files in case there are a few things in the future with settings.
The fonts
folder contains fonts that can be used, for example, in Matplotlib. Personally, I liked Roboto and PTSerif.
In addition to the apps
, conf
and fonts
folders, you can create other folders as you like. For example, the temp
folder for temporary files.
Open a command prompt ( + R →
cmd.exe
→ Enter) and enter *:
C:\Dev\Miniconda3\Scripts\conda.exe create -p C:\Dev\Jupyter\dist\pyenv3.7-win64 --copy --yes python=3 conda
* To install more recent versions of libraries, you can connect the channel conda-forge
through the argument -c conda-forge
:
C:\Dev\Miniconda3\Scripts\conda.exe create -p C:\Dev\Jupyter\dist\pyenv3.7-win64 --copy --yes -c conda-forge python=3 conda
If you later need to delete the conda-forge
channel, go to the %userprofile%
folder in the Explorer, find the .condarc
file in it, open it with a notepad and delete the line conda-forge
.
Consider this command. First comes the full path to conda.exe
. If during the installation of Minconda you checked the “Add Anaconda to the system PATH environment variable” checkbox, instead of the full path you just need to write just conda
.
The word create
gives the command to create a new environment. The argument -p
says that this environment should be created where we specify, and not in the folder C:\Dev\Miniconda3\envs
. The example contains the full path and name of the future folder pyenv3.7-win64
(decoding: python 3.7 environment for Windows 64-bit). If your command line is open in the dist
folder or you have moved to this folder with the help of the cd
, instead of the full path, you could just write pyenv3.7-win64
.
The --copy
argument tells conda
that the packages themselves must be installed in the virtual environment. Otherwise, the package will be installed in the folder C:\Dev\Miniconda3
, and in the virtual environment there will be a link to it. You will not notice this substitution until you try to start a virtual environment on another computer.
Next is the enumeration of packages. First of all, we have to install the third version of Python itself. I also indicate the conda
. Those. The conda
program will be installed twice: in Miniconda and in a virtual environment. Installing a conda
in a virtual environment increases its size quite a bit, but will allow the user to update packages in a virtual environment on a computer where Miniconda is not installed. This makes the virtual environment completely autonomous. You can even uninstall Miniconda after creating a virtual environment, and it will continue to work as if nothing had happened. I, however, leave Miniconda in case some application needs Python.
In general, besides Python and conda, you could immediately specify the necessary packages, but in 2018 I stopped doing this and instead began using pip
to install packages. First, the latest versions of pip
began to download .whl
files with already compiled libraries, and the problems with installing a number of libraries disappeared. Secondly, the size of the virtual environment when installing packages via pip
is 3 times smaller than when installing packages through conda
.
HTTP 000 CONNECTION FAILED
error when creating a virtual environmentOne of the users when executing the command
C:\Dev\Miniconda3\Scripts\conda.exe create -p C:\Dev\Jupyter\dist\pyenv3.7-win64 --copy --yes -c conda-forge python=3 conda
encountered the following error:
> C:\Users\Asus>C:\Dev\Miniconda3\Scripts\conda.exe create -p C:\Dev\Jupyter\dist\pyenv3.7-win64 --copy --yes -c conda-forge python=3 conda Collecting package metadata: failed CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/win-64/repodata.json> Elapsed: - An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(MaxRetryError('HTTPSConnectionPool(host=\'conda.anaconda.org\', port=443): Max retries exceeded with url: /conda-forge/win-64/repodata.json (Caused by SSLError("Can\'t connect to HTTPS URL because the SSL module is not available."))'))
It took me more than one hour to deal with it, because at first glance the problem is either with an incorrect installation of Miniconda or with the network. Some corporate users actually blocked this resource, but the problem occurred with the user at home. Reinstalling Miniconda did not help.
As a result, it turned out that this error means that conda.exe
did not find the openssl.exe
file. As a result, the following solution was applied:
Create a folder C:\Dev\openssl
.
In the folder C:\Dev\Miniconda3\pkgs
found a folder whose name starts with openssl
. For example, openssl-1.1.1a-he774522_0
. If there are several folders, select the one with the higher number in the title.
In the found folder, look for the openssl.exe
file and copy openssl.exe
and all the files and folders that lie with openssl.exe
in C:\Dev\openssl
.
In Windows Explorer, go to "This computer" (which lists all the drives on the computer). In the free space, right-click the mouse to open the context menu and select the “Properties” item at the very bottom.
In the window that opens, we find "Advanced system settings":
On the Advanced tab, we find the Environment Variables button:
For Windows 7 and 8: in the “Environment Variables for User” section, double click on the Path
variable. If there is no semicolon at the end of the line, put it, and at the end of this line we add:
C:\Dev\openssl;
For Windows 10: in the section “Environment Variables for a User”, double click on the Path
variable. As a result, this window should appear:
Click the "Create" button and paste the path C:\Dev\openssl
.
Close and open the command prompt again. Now everything should work. If it doesn’t work, you have to google a mistake or go to the forums.
When the creation of the virtual environment is finished, the window will look like this:
After creating the virtual environment, install the packages via pip
. First you need to activate the virtual environment. To do this, in the command window, enter:
C:\Dev\Jupyter\dist\pyenv3.7-win64\Scripts\activate
As a result, you should get something like this:
The word (base)
at the beginning of the line indicates that we have entered the virtual environment we need.
Now you can install the packages *:
pip --no-cache-dir install numpy scipy matplotlib jupyter jupyterlab
The --no-cache-dir
argument tells pip
not to cache downloaded packages. This will allow us not to increase the size of the virtual environment folder.
* There is an Intel-developed MKL (Math Kernel Library) library (en) that speeds up work on big data for popular Python libraries, in particular, Numpy and Scipy. If you want to install Numpy and Scipy that use MKL, you should use (en) intel-numpy
instead of numpy
and intel-scipy
instead of scipy
:
pip --no-cache-dir install intel-numpy intel-scipy matplotlib jupyter jupyterlab
I managed to install intel-numpy
and intel-scipy
only in a virtual environment with Python 3.6. If you want to use Numpy and Scipy with MKL in an environment with Python 3.7, you need to use the command:
conda install numpy scipy
If you are not sure what to bet, just use numpy
and scipy
.
If the installation process fails with pip
, try installing problem packages with conda
. Example:
conda install numpy scipy matplotlib jupyter jupyterlab
После того, как установка завершена, необходимо выйти из виртуального окружения. Для этого в командной строке наберите*:
conda.bat deactivate
*Раньше я набирал просто deactivate
, но это почему-то устарело, и надо набирать conda.bat deactivate
. Даже conda deactivate
будет неправильно.
Создадим несколько .bat
файлов, которые будут заставят Jupyter и Matplotlib хранить настройки в папке dist\config
, а также будут управлять запуском Jupyter Notebook и Jupyter Lab.
Каталоги размещения настроек определяются переменными среды Windows. Изменив эти переменные, мы заставим Jupyter и Matplotlib хранить свои файлы там, где это нужно именно нам. В папке C:\Dev\Jupyter\dist
создайте файл setenv.bat
следующего содержания:
@echo off set conf_path=%~dp0\conf set JUPYTER_CONFIG_DIR=%conf_path%\jupyter set JUPYTER_DATA_DIR=%conf_path%\jupyter\data set JUPYTER_RUNTIME_DIR=%conf_path%\jupyter\data\runtime set IPYTHONDIR=%conf_path%\ipython set MPLCONFIGDIR=%conf_path%\matplotlib REM Matplotlib search FFMPEG in PATH variable only! set PATH=%~dp0\apps\ffmpeg\bin;%PATH%
Разберём, что делается в этом файле.
Команда @echo off
необходима для того, чтобы в командной строке не выводилось сообщение при выполнении каждой строки нашего файла.
Команда set
создаёт переменную. Конструкция %~dp0
означает полный путь к setenv.bat
. Обратите внимание, что пробелов до и после знака =
быть не должно.
Затем мы настраиваем переменные для Jupyter:
JUPYTER_CONFIG_DIR
— папка для файлов конфигурации Jupyter ( документация (en) ),JUPYTER_DATA_DIR
— папка для устанавливаемых файлов данных (расширения и ядра ( kernel
) для Jupyter) ( документация (en) ),JUPYTER_RUNTIME_DIR
— папка для исполняемых файлов Jupyter ( runtime files
) ( документация (en) ),IPYTHONDIR
— папка для файлов конфигурации IPython ( документация (en) ),MPLCONFIGDIR
— папка, где Matplotlib хранит свои настройки и кэш ( документация (en) ).Если вы планируете создавать анимации с Matplotlib, вам понадобится FFMPEG (ru) . Я скачиваю (en) zip архив FFMPEG, распаковываю его содержание C:\Dev\Jupyter\dist\apps\ffmpeg
.
Строка, которая начинается с REM
— комментарий. Matplotlib почему-то ищет FFMPEG только в %PATH%
. Я записываю путь к FFMPEG в начало %PATH
, а не в его конец, чтобы при поиске первым нашёлся тот FFMPEG, который я положил в dist\apps
.
В папке C:\Dev\Jupyter\dist
создайте файл run_jupyter_notebook.bat
следующего содержания:
@echo off call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter-notebook.exe --notebook-dir=%1
Аналогично, в папке C:\Dev\Jupyter\dist
создайте файл run_jupyter_lab.bat
следующего содержания:
@echo off call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter-lab.exe --notebook-dir=%1
Каждый из этих файлов сначала выполняет setenv.bat
, т.е. настраивает переменные окружения, потом запускает Jupyte Notebook или Jupyter Lab и указывает ему, где папка с нашими файлами для проекта.
Предположим, что есть папка D:\my-projects
, в которой мы будем хранить файлы Jupyter Notebook или Lab. В этой папке создайте ярлыки на файлы run_jupyter_notebook.bat
и run_jupyter_lab.bat
. После этого в обязательном порядке откройте свойства каждого из этих ярлыков и сделайте пустой строку «Рабочая папка». Если вы этого не сделаете — Jupyter не увидит вашу папку!
После того, как это сделали, можете кликнуть дважды по любому из ярлыков. Сначала появится новое окно командной строки, потом откроется браузер по умолчанию и в нём запустится Jupyter Notebook или Lab в новой вкладке. Поздравляю: квест пройден!
Для Jupyter Notebook написаны расширения (о них будет подробнее в части 2). Но их недостаточно установить. Их ещё надо активировать. Согласно документации , вам нужно сделать следующее (не выполняйте эту команду!):
jupyter nbextension enable <nbextension require path>
Но мы не можем выполнить команду в таком виде, потому что настройки окажутся вне портативной сборки. Мы должны сделать иначе:
C:\Dev\Jupyter\dist\setenv.bat C:\Dev\Jupyter\dist\pyenv3.7-win64\Scripts\jupyter.exe nbextension enable <nbextension require path>
Чтобы упростить себе задачу, мы можем в папке C:\Dev\Jupyter\dist
создать файл enable_extension.bat
следующего содержания:
@echo off REM Enable extension in Jupyter Notebook. REM Example: REM enable_extension.bat widgetsnbextension call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter-nbextension.exe enable %1
В итоге наша запись в окне командной строки сократится и станет такой:
C:\Dev\Jupyter\dist\enable_extension.bat <nbextension require path>
Если вам время от времени в окне команд нужно запускать различные действия с jupyter
, можно создать в папке C:\Dev\Jupyter\dist
файл jupyter.bat
следующего содержания:
@echo off call %~dp0\setenv.bat call %~dp0\pyenv3.7-win64\Scripts\jupyter.exe %1 %2 %3 %4 %5 %6 %7 %8 %9
Аналогично можно сделать для запуска IPython
и других случаев.
Часть 1 подошла к концу. Мы создали полностью портативную и переносимую сборку Jupyter и можем работать с файлами, которые лежат в нужной нам папке. Для этого создаём ярлыки на run_jupyter_notebook.bat
и run_jupyter_lab.bat
, в свойствах ярлыков обязательно очищаем строку Рабочая папка», и всё готово к работе.
В части 2 будут рассмотрены различные вопросы кастомизации Jupyter Notebook, Jupyter Lab и Matplotlib. Научимся подключать расширения в Jupyter Notebook и Jupyter Lab, настраивать сочетания клавиш, размер шрифта и прочие настройки, напишем свои «магические команды».
Если у вас остались вопросы, но нет аккаунта на Хабре, вот мои контакты:
Telegram: https://t.me/sindzicat
E-mail: sh.andr.gen@yandex.ru
Source: https://habr.com/ru/post/438934/