Environment Modules是如何工作的？

前言

Environment Modules在我们芯片研发中已经耳熟能详了，那它是如何工作的呢？ 很多人对这个比较陌生，常常会问为什么我一个 .cshrc 只能够在 c shell 里 source，而一个 modulefile 例如 modulename 能够在多种 shell 环境中使用。

目的

本文通过对一个已经初始化好的 Environment Modules 环境的 module load 步骤分析，来看它是如何工作的。

解析

bash 环境

我们从命令开始，逐步分析一个初始化好的环境。

首先来看 module 命令，它是一个 bash 函数，调用了 bash 函数 _module_raw：

[centos@computing-server-133 ~]$ echo $0
-bash
[centos@computing-server-133 ~]$ type module
module is a function
module () 
{ 
    local _mlredir=1;
    if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
        if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
            _mlredir=0;
        else
            if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
                _mlredir=1;
            fi;
        fi;
    fi;
    case " $@ " in 
        *' --no-redirect '*)
            _mlredir=0
        ;;
        *' --redirect '*)
            _mlredir=1;
        ;;
    esac;
    if [ $_mlredir -eq 0 ]; then
        _module_raw "$@";
    else
        _module_raw "$@" 2>&1;
    fi
}

_module_raw 也是 bash 函数，它调用了 tclsh 程序，读取 tcl 脚本以及参数，生成更改环境的语句，再将其作为 eval 的参数，对当前 shell 做出更改。

[centos@computing-server-133 ~]$ type _module_raw
_module_raw is a function
_module_raw () 
{ 
    eval "$(/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh '/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/libexec/modulecmd.tcl' bash "$@")";
    _mlstatus=$?;
    return $_mlstatus
}

我们以加载 tcl-8.6.12-gcc-12.2.0-5rkwrb3 这个 modulefile 作为示例：

[centos@computing-server-133 ~]$ ml show tcl-8.6.12-gcc-12.2.0-5rkwrb3
-------------------------------------------------------------------
/nfs/tools/os/spack/share/spack/modules/linux-centos7-x86_64_v4/tcl-8.6.12-gcc-12.2.0-5rkwrb3:

module-whatis   {Tcl (Tool Command Language) is a very powerful but easy to learn dynamic programming language, suitable for a very wide range of uses, including web and desktop applications, networking, administration, testing and many more. Open source and business-friendly, Tcl is a mature yet evolving language that is truly cross platform, easily deployed and highly extensible.}
module          load zlib-1.2.13-gcc-12.2.0-pobtkn4
prepend-path    --delim : PATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin
prepend-path    --delim : MANPATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/man
prepend-path    --delim : MANPATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/share/man
prepend-path    --delim : PKG_CONFIG_PATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/lib/pkgconfig
prepend-path    --delim : CMAKE_PREFIX_PATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/.
setenv          TCL_LIBRARY /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/lib/tcl8.6
-------------------------------------------------------------------[centos@computing-server-133 ~]$ ml
No Modulefiles Currently Loaded.
[centos@computing-server-133 ~]$ /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh '/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/libexec/modulecmd.tcl' bash load tcl-8.6.12-gcc-12.2.0-5rkwrb3
PATH=/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/bin:/nfs/tools/os/spack/bin:/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/etc:/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/centos/.local/bin:/home/centos/bin; export PATH;
__MODULES_SHARE_MANPATH=:1; export __MODULES_SHARE_MANPATH;
MANPATH=/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/share/man:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/man:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/zlib-1.2.13-pobtkn4lnv467woi4cerdauysxm42mn2/share/man:/opt/ibm/lsfsuite/lsf/10.1/man:; export MANPATH;
__MODULES_LMPREREQ=tcl-8.6.12-gcc-12.2.0-5rkwrb3\&zlib-1.2.13-gcc-12.2.0-pobtkn4; export __MODULES_LMPREREQ;
_LMFILES_=/nfs/tools/os/spack/share/spack/modules/linux-centos7-x86_64_v4/zlib-1.2.13-gcc-12.2.0-pobtkn4:/nfs/tools/os/spack/share/spack/modules/linux-centos7-x86_64_v4/tcl-8.6.12-gcc-12.2.0-5rkwrb3; export _LMFILES_;
LOADEDMODULES=zlib-1.2.13-gcc-12.2.0-pobtkn4:tcl-8.6.12-gcc-12.2.0-5rkwrb3; export LOADEDMODULES;
CMAKE_PREFIX_PATH=/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/.:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/zlib-1.2.13-pobtkn4lnv467woi4cerdauysxm42mn2/.; export CMAKE_PREFIX_PATH;
TCL_LIBRARY=/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/lib/tcl8.6; export TCL_LIBRARY;
__MODULES_LMTAG=zlib-1.2.13-gcc-12.2.0-pobtkn4\&auto-loaded; export __MODULES_LMTAG;
PKG_CONFIG_PATH=/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/lib/pkgconfig:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/zlib-1.2.13-pobtkn4lnv467woi4cerdauysxm42mn2/lib/pkgconfig; export PKG_CONFIG_PATH;
test 0;
[centos@computing-server-133 ~]$ which tclsh
/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh

可以看到它生成了的语句，包括了 modulefile 里显式对 PATH、MANPATH、PKG_CONFIG_PATH、CMAKE_PREFIX_PATH 等环境变量的修改，还包括了用户未定义的，对 Environment Modules 工具自身使用的变量做了修改。执行完查询后，tclsh 并没有被加载上。需要将这些语句传入 eval，才能在当前 shell 环境生效：

[centos@computing-server-133 ~]$ eval `/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh '/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/libexec/modulecmd.tcl' bash load tcl-8.6.12-gcc-12.2.0-5rkwrb3`
Loading tcl-8.6.12-gcc-12.2.0-5rkwrb3
  Loading requirement: zlib-1.2.13-gcc-12.2.0-pobtkn4
[centos@computing-server-133 ~]$ which tclsh
/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh

c shell 环境

在 csh 环境中，module 是一个 alias，同样地，它将生成的更改环境的语句传入 eval，使得环境修改生效。

[centos@computing-server-133 ~]$ echo $0
csh
[centos@computing-server-133 ~]$ which module
module: 	 aliased to set _prompt=$prompt:q; set prompt=""; eval "`/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh '/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/libexec/modulecmd.tcl' csh !*:q`"; set _exit="$status"; set prompt=$_prompt:q; unset _prompt; test 0 = $_exit

同样的，我们加载 tcl-8.6.12-gcc-12.2.0-5rkwrb3 这个 modulefile：

[centos@computing-server-133 ~]$ /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh '/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/libexec/modulecmd.tcl' csh load tcl-8.6.12-gcc-12.2.0-5rkwrb3
setenv PATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin:/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/etc:/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/bin:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/environment-modules-5.2.0-upsh6wzxsg7airzqpjayrvoapfrtgttp/bin:/nfs/tools/os/spack/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/centos/.local/bin:/home/centos/bin;
setenv __MODULES_SHARE_MANPATH :1;
setenv MANPATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/share/man:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/man:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/zlib-1.2.13-pobtkn4lnv467woi4cerdauysxm42mn2/share/man:/opt/ibm/lsfsuite/lsf/10.1/man:;
setenv __MODULES_LMPREREQ tcl-8.6.12-gcc-12.2.0-5rkwrb3\&zlib-1.2.13-gcc-12.2.0-pobtkn4;
setenv _LMFILES_ /nfs/tools/os/spack/share/spack/modules/linux-centos7-x86_64_v4/zlib-1.2.13-gcc-12.2.0-pobtkn4:/nfs/tools/os/spack/share/spack/modules/linux-centos7-x86_64_v4/tcl-8.6.12-gcc-12.2.0-5rkwrb3;
setenv LOADEDMODULES zlib-1.2.13-gcc-12.2.0-pobtkn4:tcl-8.6.12-gcc-12.2.0-5rkwrb3;
setenv CMAKE_PREFIX_PATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5

cctyigmfm37/.:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/zlib-1.2.13-pobtkn4lnv467woi4cerdauysxm42mn2/.;
setenv TCL_LIBRARY /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/lib/tcl8.6;
setenv __MODULES_LMTAG zlib-1.2.13-gcc-12.2.0-pobtkn4\&auto-loaded;
setenv PKG_CONFIG_PATH /nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/lib/pkgconfig:/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/zlib-1.2.13-pobtkn4lnv467woi4cerdauysxm42mn2/lib/pkgconfig;
test 0;
Loading tcl-8.6.12-gcc-12.2.0-5rkwrb3
  Loading requirement: zlib-1.2.13-gcc-12.2.0-pobtkn4
[centos@computing-server-133 ~]$ which tclsh
/nfs/tools/os/spack/opt/spack/linux-centos7-x86_64_v4/gcc-12.2.0/tcl-8.6.12-5rkwrb3btdklgrkshepg5cctyigmfm37/bin/tclsh

总结

上述介绍了 Environment Modules 对环境的修改原理。主要是通过对 tcl 语句的 modulefile 解析，生成对应 shell 类型的修改语句，再通过 eval 语句来完成对当前 shell 的环境修改。

真正难得的是它如何生成对环境修改的语句。后续我们将深入探讨如何解析 modulefile 并生成环境更改语句。

TODO

如何解析 modulefile 并生成环境更改语句。