Java程序辅导
C C++ Java Python Processing编程在线培训 程序编写 软件开发 视频讲解
C C++ Java Python Processing编程在线培训 程序编写 软件开发 视频讲解
Virtual Environments on Lewis and Clark clusters Research Computing Support Services (RCSS) http://docs.rnet.missouri.edu RCSS CIE team Ashkan Mirzaee, Asif Magdoom Brian Marxkors, Christina Roberts Predrag Lazic • Conda is an open-source package management system and environment management system • Conda quickly installs, runs and updates packages and their dependencies • Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN, and more MINICONDA • A free minimal installer for Conda • It is a small, bootstrap version of Anaconda and includes only Conda, Python, and a small number of other useful packages Why using Conda? Conda let us to create a virtual environment (env) to run a specific software including all the dependencies. Creator of the env has permissions to edit the environment without sudo privilege. Packages Applications will often use packages and modules that don’t come as part of the standard library Conda channels are the locations where packages are stored: main: default channel and usually has the most stable version of packages conda-forge: community channel made up and usually has the latest version of packages r: includes R packages bioconda: distribution of bioinformatics software etc. Find Conda pkgs Go https://anaconda.org/ and search packages … Conda configuration By default, Conda stores environments and packages within the folder ~/.conda. To avoid using up all your home folder's quota, which can easily happen when using Conda, we recommend placing the following within the file ~/.condarc : Copy the file by running: cp /group/training/env-intro/.condarc ~ Now let's initialize Conda for bash shell interaction: envs_dirs: - /data/$USER/miniconda/envs pkgs_dirs: - /data/$USER/miniconda/pkgs auto_activate_base: false channels: - defaults - conda-forge Request resources and load Miniconda3 srun -p Interactive --qos interactive --mem 16G --pty bash module load miniconda3 conda info conda init bash source .bashrc p ri o ri ty Conda commands Let's create a Python environment including pandas, numpy and sqlite: Help conda help or conda-h Create an env conda create --name/-n ... conda create -n = = ... conda create -n --file Note: by default conda creates new envs at /home/$USER/data/miniconda/envs Create an env in a certain path conda create --prefix/-p ... Different channels conda create --name --channel/-c ... conda create -n -c -c ... List envs and pkgs conda env list conda list -n or -p # use --export/-e to save packages for future use Activate env conda/source activate or Deactivate env conda deactivate priority Interactive example Let's create a Python environment including redis and sqlite: We can add other packages to our env: Request resources and load Miniconda3 srun -p Interactive --qos interactive --mem 16G --pty bash module load miniconda3 conda info conda create -n mypy-env python sqlite redis-py conda activate mypy-env conda list python > import redis > import sqlite3 > exit() Make sure env is activated conda install pandas numpy python > import numpy > exit() conda deactivate # to deactivate env Batchfile example We can submit a batchfile to create an env. For instance, we can create an R env including lme4 by: #!/bin/bash #SBATCH --partition Lewis #SBATCH --mem 16G module load miniconda3 if [ ! -d ~/data/miniconda/myr-env ]; then # creates the env if not exist conda create --yes -n myr-env -c conda-forge r-base r-lme4 fi sources activate myr-env Rscript test.R #!/usr/bin/r library(lme4) fm = lmer(Reaction ~ Days + (Days | Subject), sleepstudy) summary(fm) cp -r /group/training/env-intro/ ~ && cd ~/env-intro # copy files sbatch batch-example.sh # submit batchfile sbatch jupyter-env.sh b at ch -e xa m p le .s h te st .R Sharing envs We can share our recipes: And regenerate the same env by: We can create an env in a shared location so that anyone who access the location can use it: For example: conda list -n or -p -e > requirements.txt eg. conda list -n mypy-env -e > requirements.txt Note: we can activate the env and only run "conda list -e" conda create -p ... conda create -n or -p --file requirements.txt eg. conda create -n mypy-cp-env --file requirements.txt #!/bin/bash #SBATCH --partition Lewis #SBATCH --mem 16G module load miniconda3 if [ ! -d /group/path ]; then conda create --yes -p /group/path -c conda-forge r-base r-lme4 r-sf fi Removing envs We can remove envs by: Also, we can remove downloaded packages (caches) from ~/data/miniconda/pkgsby: Make sure env is deactivated conda deactivate conda env list codna env remove -n or -p eg. conda env remove -n mypy-env conda env remove -p ~/data/miniconda/myr-env cd ~/data/miniconda/ rm -r pkgs/ To clean all caches conda clean --all Jupyter First need Jupyter: Request resources and load Miniconda3 srun -p Interactive --qos interactive --mem 16G --pty bash module load miniconda3 Create the env conda create -n jupyter-env jupyterlab matplotlib numpy Activate the env conda activate jupyter-env Start jupyter-lab jupyter-lab --ip 0.0.0.0 Or start jupyter-notebook jupyter-notebook --ip 0.0.0.0 HOST:PORT URL Jupyter (cont.) Open a terminal in your local computer: Now Ctrl+click on the URL to open Jupyter server on your browser. Need a local port forwarding ssh -NfL port:host:port user@lewis.rnet.missouri.edu eg. ssh -NfL 8888:lewis-r7425-htc5:8888 user@lewis.rnet.missouri.edu Jupyter for R We can also use Jupyter as an IDE for R After running the local port forwarding, Ctrl+click on the URL to open Jupyter server on your browser. Activate Jupyter env conda install -c conda-forge r-base=4.0.3 r-irkernel jupyter-lab --ip 0.0.0.0 What is next: RCSS Documentation http://docs.rnet.missouri.edu XSEDE https://www.xsede.org/for-users/training Software Carpentry https://software-carpentry.org/lessons/ HPC Carpentry https://www.hpc-carpentry.org/ Data Carpentry https://datacarpentry.org/lessons/ Cornell Virtual Workshop https://cvw.cac.cornell.edu/topics Pittsburgh Supercomputing Center (PSC) https://www.psc.edu/resources-for-users/training/ TACC Learning Portal https://learn.tacc.utexas.edu/course/ Linux intro http://docs.rnet.missouri.edu/software/linux-intro/ Cluster intro http://docs.rnet.missouri.edu/lewis-and-clark-clusters/ Version control • Git https://git-scm.com/book/en/v2 Job dependencies • Slurm dependency option (--dependency) https://slurm.schedmd.com/sbatch.html • Snakemake https://snakemake.readthedocs.io/ Parallel programming • Scientific languages C/Fortran − OpenMP − OpenACC for GPU parallelization − MPI for massive parallelization • Python − NumPy − Numba − mpi4py Feedback and Questions mudoitrcss@missouri.edu