CESM2 实验笔记

一：关于重启、运行方式（重启文件、初始文件）

1. 重新启动 RESTART

参考：
https://bb.cgd.ucar.edu/cesm/threads/how-to-restart-cesm2-1-3-fhist-from-a-backup.5245/#post-36275

适用情况：模型运行因时间不足或其它原因中断，需要继续运行，有restart文件。
方法1：将CONTINUE_RUN改为TRUE，其它保持不变。

./xmlchange CONTINUE_RUN=TRUE

If you are trying to simply continue a stopped run, then all you should set is CONTINUE_RUN to TRUE, while leaving everything else as it was originally.

注意：/run目录下的rpointer文件中的日期要与重启文件的日期一致。

方法2：
可以新建一个 hybrid 或者branch run。需要设置相应的日期变量。但是CONTINUE_RUN应该设为FALSE。因为hybrid或者branch run是一个新的拟合案例。

If, however, you are trying to create a new hybrid run, as your RUN_TYPE indicates, then you will want to modify all of the date variables like you have done, but set CONTINUE_RUN to FALSE, as a hybrid (or branch) run will be considered a new simulation.

1.1 关于重启文件

每个活动组件和部分数据组件会定期地写入重启文件，间隔时长由env_run.xml中的$REST_OPTION 和 $REST_N指定。重启文件允许模型停下来然后在重新开始，就像没有停下来过一样。所有的组件都会从驱动处接收重启和停止信息，以及通过驱动写入指定的重启和停止文件。
Restart files are written by each active component (and some data components) at intervals dictated by the driver via the setting of the env_run.xml variables, $REST_OPTION and $REST_N. Restart files allow the model to stop and then start again with bit-for-bit exact capability (i.e. the model output is exactly the same as if it had never been stopped). The driver coordinates the writing of restart files as well as the time evolution of the model. All components receive restart and stop information from the driver and write restarts or stop as specified by the driver.

组件写入重启文件的时候，也会写入一个重启指向文件a restart pointer file，形式为rpointer.$component。重启指向文件包含了重启文件的目录。
Whenever a component writes a restart file, it also writes a restart pointer file of the form, rpointer.$component. The restart pointer file contains the restart filename that was just written by the component. Upon a restart, each component reads its restart pointer file to determine the filename(s) to read in order to continue the model run. As examples, the following pointer files will be created for a component set using full active model components.

• rpointer.atm
• rpointer.drv
• rpointer.ice
• rpointer.lnd
• rpointer.rof
• rpointer.cism
• rpointer.ocn.ovf
• rpointer.ocn.restart

如果开启了短期归档，重启数据和指向文件会在$DOUT_S_ROOT/rest/yyyy-mm-dd-sssss目录下。

If short-term archiving is turned on, then the model archives the component restart datasets and pointer files into $DOUT_S_ROOT/rest/yyyy-mm-dd-sssss, where yyyy-mm-dd-sssss is the model date at the time of the restart (see below for more details).
If long-term archiving these restart then archived in $DOUT_L_MSROOT/rest. DOUT_S_ROOT and DOUT_L_MSROOT are set in env_run.xml, and can be changed at any time during the run.

2. RUN_TYPE 初始化运行方式

参考：https://bb.cgd.ucar.edu/cesm/threads/suggestions-for-modifying-the-start-date-cesm2.4921/#post-35058

case的初始化类型可通过env_run.xml文件中的 $RUN_TYPE 变量设置。初始化类型共有三种：

• startup

In a startup run (the default), all components are initialized using baseline states. These states are set independently by each component and can include the use of restart files, initial files, external observed data files, or internal initialization (that is, a “cold start”). In a startup run, the coupler sends the start date to the components at initialization. In addition, the coupler does not need an input data file. In a startup initialization, the ocean model does not start until the second ocean coupling step.

• branch

In a branch run, all components are initialized using a consistent set of restart files from a previous run (determined by the $RUN_REFCASE and $RUN_REFDATE variables in env_run.xml). The case name generally is changed for a branch run, but it does not have to be. In a branch run, the $RUN_STARTDATE setting is ignored because the model components obtain the start date from their restart data sets. Therefore, the start date cannot be changed for a branch run. This is the same mechanism that is used for performing a restart run (where $CONTINUE_RUN is set to TRUE in the env_run.xml file). Branch runs typically are used when sensitivity or parameter studies are required, or when settings for history file output streams need to be modified while still maintaining bit-for-bit reproducibility. Under this scenario, the new case is able to produce an exact bit-for-bit restart in the same manner as a continuation run if no source code or component namelist inputs are modified. All models use restart files to perform this type of run. $RUN_REFCASE and $RUN_REFDATE are required for branch runs. To set up a branch run, locate the restart tar file or restart directory for $RUN_REFCASE and $RUN_REFDATE from a previous run, then place those files in the $RUNDIR directory. See setting up a branch run.

• hybrid

A hybrid run is initialized like a startup but it uses initialization data sets from a previous case. It is similar to a branch run with relaxed restart constraints. A hybrid run allows users to bring together combinations of initial/restart files from a previous case (specified by $RUN_REFCASE) at a given model output date (specified by $RUN_REFDATE). Unlike a branch run, the starting date of a hybrid run (specified by $RUN_STARTDATE) can be modified relative to the reference case. In a hybrid run, the model does not continue in a bit-for-bit fashion with respect to the reference case. The resulting climate, however, should be continuous provided that no model source code or namelists are changed in the hybrid run. In a hybrid initialization, the ocean model does not start until the second ocean coupling step, and the coupler does a “cold start” without a restart file.

• hybrid vs. branch
  如果我们要跑一个和控制实验（control run）仅有些微差别的实验，比如我们只是想做一些微小调整，增加历史输出（history output)，或者从一个CESM spun-up的初始态来开始模拟并保持精确的重新启动（restart）（这与重启是一致的），在这些情况下，hybrid和branch runs非常有用。
  使用hybrid run: 对于大多数不需要逐位重新启动(bit for bit restart)的应用。可以为模式运行指定新的开始日期。
  使用branch run:仅适用于需要完全重新启动的应用。不能为模型运行指定新的开始日期。它将由参考案例分配（run_refdate）。

• GET_REFCASE: default=TRUE；如果是TRUE，数据需要预先暂存在可执行目录中（这与去年的CESM2.0不同）。

相关参数/变量

• RUN_TYPE 初始化类型
• RUN_REFCASE 参考案例
• RUN_REFDATE 参考案例的运行结束时间
• RUN_STARTDATE 当前实际案例的运行开始时间
• GET_REFCASE TRUE or FALSE 是否自动copy重启文件

startup

重新预热，模型从零开始运行，需要经历较长时间达到平衡状态。

• RUN_TYPE=startup
• RUN_REFCASE：无效
• RUN_REFDATE：无效
• RUN_STARTDATE：可以任意指定，例如1850-01-01，或者1901-01-01
• GET_REFCASE=FALSE

branch

• RUN_TYPE=branch
• RUN_REFCASE=自己指定
• RUN_REFDATE=自己指定，参考案例的运行结束日期
• RUN_STARTDATE=无效，branch run 会自动从重启文件中读取运行开始时间，只能接着重启文件接着跑，由参考案例分配（run_refdate）
• GET_REFCASE=FALSE

hybrid

• RUN_TYPE=hybrid
• RUN_REFCASE=自己指定
• RUN_REFDATE=自己指定，参考案例的运行结束日期
• RUN_STARTDATE=自己指定，但和重启文件的日期应当是一致的，即RUN_STARTDATE=RUN_REFDATE
• GET_REFCASE=FALSE

注意：实际上，无论是branch还是hybrid，RUN_STARTDATE和RUN_REFDATE原理上始终是相等的，尽管branch中无需设置RUN_STARTDATE。
重启文件中的日期必须要和设置的运行开始日期RUN_STARTDATE相等，否则便会报错。
要想branc或hybrid运行，必须要提供相应日期的重启文件才可。

# 查询
./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASE./xmlquery STOP_OPTION,STOP_N,RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RESUBMIT,RUN_STARTDATE,RUN_REFDIR,DOUT_S,DOUT_S_ROOT,JOB_WALLCLOCK_TIME,GET_REFCASE# 修改
./xmlchange RUN_TYPE=startup,RUN_REFCASE=case.std,RUN_REFDATE=0001-01-01,RUN_STARTDATE=1850-01-01,GET_REFCASE=FALSE

3. branch 或者 hybrid运行的示例

CESM1.1 用户指南：How do I set up a branch or hybrid run?

应用：已有案例B1850用于预热20年，新建案例BHIST用于历史输出1901-2000年，利用B1850的预热结果

第一步：newcase BHIST （克隆案例是一个更好的选择，能够保证设置一模一样，避免二次设置）
./careate_newcase --compset=BHIST --res=f09_g17 --case ~/CESM2/scratch/BHIST

第二步：修改env_run.xml文件中的运行方式
for branch run:

./xmlchange RUN_TYPE=branch
./xmlchange RUN_REFCASE=B1850
./xmlchange RUN_REFDATE=0001-02-01    # 视情况而定，一般选用最近的重启文件的日期 
./xmlchange GET_REFCASE=FALSE

for a branch run, BHIST案例的env_run.xml文件应该和B1850案例的env_run.xml文件一模一样，除了RUN_TYPE设置外。在案例B1850中user_nl_xxx所做的任何修改应该在案例BHIST中user_nl_xxx一致。
branch run 中， RUN_STARTDATE 这一参数会被忽略.

for hybird run:

./xmlchange RUN_TYPE=hybird
./xmlchange RUN_REFCASE=B1850
./xmlchange RUN_REFDATE=0001-02-01    # 视情况而定，一般选用最近的重启文件的日期 
./xmlchange GET_REFCASE=FALSE

hybird run 中，开始模拟的日期可以用 RUN_STARTDATE 设置。
第三步：setup and build

./case.setup
./case.build --skip-provenance-check &> FHIST_BGC.build.spinup.log

第四步：复制初始和重启文件
如果想接着0001-02-01后跑，那么把B1850中的0001-02-01初始和重启文件复制到案例BHIST的/run/目录下。

cp -r /BIGDATA1/sysu_syli_3/CESM2/scratch/B1850test/archive/rest/0001-01-03-00000/* /BIGDATA1/sysu_syli_3/CESM2/scratch/BHIST/run/    # 复制，将0001-01-03-00000文件夹下的所有文件复制到run/目录下。
# or =
cp -r /BIGDATA1/sysu_syli_3/CESM2/scratch/B1850test/archive/rest/0001-01-03-00000/* /BIGDATA1/sysu_syli_3/CESM2/scratch/BHIST/run
# 或者
./xmlchange GET_REFCASE=TRUE  # 是否会自动复制重启文件，会在inputdata/init2中新建一个FHIST_BGC目录，将文件拷贝至改目录，随后在caseroot/run/目录中创建rpointer文件，指向init2/FHIST_BGC/目录
# 但问题是天河2没网，没办法拷贝

第五步：设置运行时长，短期归档，CPU时长等

./xmlchange DOUT_S=TRUE
./xmlchange STOP_N=1,STOP_OPTION=nyears
./xmlchange JOB_WALLCLOCK_TIME=12:00:00   # 依据模型运行速度而定，含义是一次提交最多运行12小时。如果在设定的时间内模型没有跑完，会报超时错误

第六步：submit

./case.submit

第七步：resubmit

./xmlchange CONTINUE_RUN=TRUE
./xmlchange RESUBMIT=10   # 再次重复提交10次，一次一年，则共运行11年
./case.submit

4. 其他声明

• 控制实验
  控制实验：驱动不随时间变化，不断重复。例如B1850，用1850年的驱动不断循环重复运行，直至模型达到平衡状态，称之为工业化前的控制实验（a pre-industrial control run）。
  历史实验：驱动随时间变化。例如BHIST，利用1850-2015年的驱动运行模型，输出1850-2015年的历史输出。

b.e21.B1850.f09_g17.CMIP6-piControl.001
B1850的控制实验，会一致重复运行1850年。如果想获取2000年或其他年份的重启文件，应当选用BHIST实验。
When we run a B1850, it cycles over pre-industrial conditions. Basically, the model perpetually repeats the year 1850.
So the year 0151 doesn’t correspond to year 2000. To get the restart files representative of year 2000, you need to get restart files from a BHIST and not a B1850.

建议：控制实验的运行时间RUN_REFDATE只是实验运行时间的一个计时器，与我们使用的日历没有关系。

对于控制实验（B1850），你可以指定任意一年作为模型运行的开始时间。但是你必须有相应时间的重启文件。
如果想要修改模型运行的开始时间，请在build之前修改。

• 关于CONTINUE_RUN
  对于初次运行而言，CONTINUE_RUN始终为FALSE。
  如果你想要的继续运行，CONTINUE_RUN才设为为TRUE。

5. 关于RESUBMIT

长时间运行，一般超算的作业提交系统有单次最大运行时间限制，可以把100年的运行分10次提交，每次运行10年，不超过超算的时间限制。
当RESUBMIT>0，CONTINUE_RUN会自动设为TRUE。

例如，
STOP_OPTION=nyears
STOP_N=1
RESUBMIT=9
表明一共运行10年，包括了初始运行的1年。

二：关闭FHIST_BGC实验的大气化学模块

大气化学模块

./xmlquery CAM_CONFIG_OPTSCAM_CONFIG_OPTS: -phys cam6CAM_CONFIG_OPTS: -phys cam6 -chem none./xmlchange --append CAM_CONFIG_OPTS="-chem none"
./xmlchange CAM_CONFIG_OPTS="-phys cam6 -chem none"
./xmlchange CAM_CONFIG_OPTS="-phys cam5 -chem none"

三：PE设置——env_mach_pes.xml文件的修改

参考：https://wenku.baidu.com/view/bae03427cdbff121dd36a32d7375a417866fc11b.html

调用的CPU核数：

• 并行： 并行模块调用的CPU核心数的总和
• 串行：串行模块调用的CPU核心数的最大值

Changing the PE layout
env_mach_pes.xml variables determine the number of processors for each component, the number of instances of each component and the layout of the components across the hardware processors.
There are NTASKS, NTHRDS, and ROOTPE input variables for every component in env_mach_pes.xml.
The total number of hardware processors allocated to a component is NTASKS * NTHRDS.
If all components have identical NTASKS, NTHRDS, and ROOTPE set, all components will run sequentially on the same hardware processors.

• env_mach_pes.xml
  Sets component machine-specific processor layout (see changing pe layout ). The settings in this are critical to a well-load-balanced simulation (see load balancing).
• 修改变量：ntasks,nthrds,rootpe,ninst
• 修改时间：严格来说，env_mach_pes.xml应当在setup之前修改。若setup后又对上述参数进行了更改，则需先resetup。
• 小心修改，用于分配CPU算力，运行长时间的案例，建议进行load_balance测试，以提高效率，节约成本。

./xmlquery NTASKS,NTHRDS,ROOTPE,NINST
./preview_run

解释：

• NTASKS：MPI的任务数
• NTHRDS：每个MPI任务调用的CPU核心数。
• 所以总计调用的CPU核心数为NTASKS*NTHRDS
• ROOTPE：MPI任务的全局设置，即分配哪个CPU给哪个MPI任务。串行或并行
• NINST : 组件的实例数the number of component instances (will be spread evenly across NTASKS)

例子1：

<entry id="NTASKS_ATM" value="16" />
<entry id="NTHRDS_ATM" value="4"  />
<entry id="ROOTPE_ATM" value="0"  />
<entry id="NTASKS_OCN" value="64" />
<entry id="NTHRDS_OCN" value="1"  />
<entry id="ROOTPE_OCN" value="16" />

解释：并行设置ATM和OCN
ATM调用16个MPI任务，MPI任务序号为0-15。每个MPI任务分配4个CPU核心，共调用164=64个CPU核心，CPU核心序号为0-63。
OCN调用64个MPI任务，MPI任务序号为16-79.每个MPI任务分配1个CPU核心，共调用641=64个CPU核心，CPU核心序号为64-128。
所以，共调用128个CPU核心（并行）

例子2：将ROOTPE_OCN设为64.
解释：并行设置ATM和OCN
ATM调用16个MPI任务，MPI任务序号为0-15。每个MPI任务分配4个CPU核心，共调用164=64个CPU核心，CPU核心序号为0-63。
OCN调用64个MPI任务，MPI任务序号为64-127。发现MPI任务序号16-63空闲（481=48个CPU），对应的CPU为64-111。每个MPI任务分配1个CPU核心，共调用64*1=64个CPU核心，CPU核心序号为112-176。
所以，共调用64+48+64=176个CPU核心（并行）

具体案例分析1：所有模块都是顺序运行（ROOTPE=0），单线程（NTHRDS=1），每个模块都是24个任务。即每个模块都依次调用了241=24个CPU核心。所以总计调用了一个节点。


具体案例分析2：
单线程，每个模块都有24个任务，即每个模块都调用了241=24个CPU核心。
其中，CPL、ATM、LND、ROF、GLC、WAV和ESP模块顺序运行，调用了第0至第23个CPU（24个）。ICE并行，调用了第96至第119个CPU核心（24个）。OCN并行，调用了第192至第215个CPU核心（24个）。
故，总共调用了216个CPU核心，但实际工作的只有72个CPU核心，144个核心被闲置了，真是浪费！

1. CPU核数不是越多越好，随着CPU核数增加，模型的运行速度（模式年/天）加快，运行效率增加（核时/模式年）。但当CPU核数增加到一定程度时，模型的运行速度和效率都会降低。因此，最经济的设置是CPU核数应当适中，既不能过大，也不能过小。增加核数，每个CPU负责的区域变小，运行速度加快。但核数过多，划分的区域也就越多，CPU间的数据交换花费的时间也就越多，影响总体的运行效率。
2. 各模块顺序运行的CPU配置，运行速度慢，资源消耗大，效率低，不建议使用。仅修改NTASKS，各个模块依次调用相同数量的CPU核心数。
3. 进一步分析发现大气和海洋模块对模式的运行速度影响最大。但是在CESM中陆地模块和大气模块只能顺序运算，因此，可考虑将海洋模块与其余模块并行运算，大气、陆地、海冰等其他模块顺序运算。模式的运行速度和效率和模块间的串并行方案有关。建议将消耗资源大和运行速度慢的大气和海洋模块并行。其他模块串行。
   
4. 在保证每个模块所分配的CPU核数不是计算瓶颈的前提下，适当减少海洋模块的核数反而能在一定程度上使得模式运行速度提高、资源消耗减少。对于全球1°分辨率的全耦合案例而言，海洋模块分配80-120个CPU就可以保证其运算速度和效率维持在一个高水平上。
5. 总结：影响模式运行速度和效率的因素有总的CPU核数、各个模块分配的核数、模块间的串并行策略等。

四：更改开始运行时间

目的：修改FHIST_BGC实验的开始运行时间

除改变开始运行时间外，其它设置均不变：

1. create_newcase

修改PE

./xmlquery NTASKS,NTHRDS,ROOTPE
./xmlchange NTASKS=-10
./xmlchange NTASKS_ESP=10

2. setup

./case.setup 

修改user_nl_clm

vim user_nl_clmhist_empty_htapes = .true 
hist_avgflag_pertape = 'A'
hist_nhtfrq = 0  
hist_mfilt = 1
hist_fincl1="QOVER","QH2OSFC","H2OSOI","GPP","NPP","NEP","TLAI","FLDS","FSDS","Rnet","QVEGT","QSOIL","QINTR","FSH","TSA","Q2M"

./xmlchange DOUT_S=TRUE
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange STOP_N=1,STOP_OPTION=nmonths./xmlquery STOP_OPTION,STOP_N,RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RESUBMIT,RUN_STARTDATE,RUN_REFDIR,DOUT_S,DOUT_S_ROOT,JOB_WALLCLOCK_TIME

修改运行开始时间

3. build

./case.build --skip-provenance-check &> build.log

4. submit

./case.submit &> submit.log

1. FHIST_BGC_s1-----从1850年开始跑，将运行方式从hybrid改为startup

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_s1 &> FHIST_BGC_s1.log

./xmlchange RUN_TYPE=startup,RUN_REFCASE=case.std,RUN_REFDATE=0001-01-01,RUN_STARTDATE=1850-01-01,GET_REFCASE=FALSE

# 默认
[sysu_syli_3@ln42%tianhe2-H FHIST_BGC_s1]$ ./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASE
Results in group run_begin_stop_restartRUN_TYPE: hybridCONTINUE_RUN: FALSERUN_REFCASE: b.e20.BHIST.f09_g17.20thC.297_01_v3RUN_REFDATE: 1979-01-01RUN_STARTDATE: 1979-01-01RUN_REFDIR: cesm2_initGET_REFCASE: TRUE# 修改为
[sysu_syli_3@ln42%tianhe2-H FHIST_BGC_s1]$ ./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASEResults in group run_begin_stop_restartRUN_TYPE: startupCONTINUE_RUN: FALSERUN_REFCASE: case.stdRUN_REFDATE: 0001-01-01RUN_STARTDATE: 1850-01-01RUN_REFDIR: cesm2_initGET_REFCASE: FALSE

2. FHIST_BGC_s2-----从1890年开始跑，将运行方式从hybrid改为startup

对比实验1与2，目的在于测试startup 运行，
RUN_REFCASE和RUN_REFDATE有没有起作用？
RUN_STARTDATE怎么起作用？

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_s2 &> FHIST_BGC_s2.log

./xmlchange RUN_TYPE=startup,RUN_STARTDATE=1890-01-01,GET_REFCASE=FALSE

# 默认
[sysu_syli_3@ln42%tianhe2-H FHIST_BGC_s2]$ ./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASEResults in group run_begin_stop_restartRUN_TYPE: hybridCONTINUE_RUN: FALSERUN_REFCASE: b.e20.BHIST.f09_g17.20thC.297_01_v3RUN_REFDATE: 1979-01-01RUN_STARTDATE: 1979-01-01RUN_REFDIR: cesm2_initGET_REFCASE: TRUE# 修改为
[sysu_syli_3@ln42%tianhe2-H FHIST_BGC_s2]$ ./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASEResults in group run_begin_stop_restartRUN_TYPE: startupCONTINUE_RUN: FALSERUN_REFCASE: b.e20.BHIST.f09_g17.20thC.297_01_v3RUN_REFDATE: 1979-01-01RUN_STARTDATE: 1890-01-01RUN_REFDIR: cesm2_initGET_REFCASE: FALSE

结论：当运行方式为startup时，RUN_REFCASE和RUN_REFDATE不起作用，最好把GET_REFCASE设为FALSE（没有测试为TRUE的情况）。可以直接更改RUN_STARTDATE为想要开始运行的日期。

3. FHIST_BGC_s3-----从1901年开始跑，将参考案例换成b.e21.B1850.f09_g17.CMIP6-piControl.001_v2/0601-01-01/

cesm2_init\b.e21.B1850.f09_g17.CMIP6-piControl.001_v2\0501-01-01参考案例目前不能用
https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/cesm2_init/b.e21.B1850.f09_g17.CMIP6-piControl.001_v2/0501-01-01/b.e21.B1850.f09_g17.CMIP6-piControl.001_v2.pop.r.0501-01-01-00000.nc没下好：

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_s3 &> FHIST_BGC_s3.log

./xmlchange RUN_STARTDATE=1901-01-01,GET_REFCASE=TRUE,RUN_REFCASE=b.e21.B1850.f09_g17.CMIP6-piControl.001_v2,RUN_REFDATE=0601-01-01

# 默认
[sysu_syli_3@ln42%tianhe2-H FHIST_BGC_s3]$ ./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASEResults in group run_begin_stop_restartRUN_TYPE: hybridCONTINUE_RUN: FALSERUN_REFCASE: b.e20.BHIST.f09_g17.20thC.297_01_v3RUN_REFDATE: 1979-01-01RUN_STARTDATE: 1979-01-01RUN_REFDIR: cesm2_initGET_REFCASE: TRUE# 修改为
[sysu_syli_3@ln42%tianhe2-H FHIST_BGC_s3]$ ./xmlquery RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RUN_STARTDATE,RUN_REFDIR,GET_REFCASEResults in group run_begin_stop_restartRUN_TYPE: hybridCONTINUE_RUN: FALSERUN_REFCASE: b.e21.B1850.f09_g17.CMIP6-piControl.001_v2RUN_REFDATE: 0601-01-01RUN_STARTDATE: 1901-01-01RUN_REFDIR: cesm2_initGET_REFCASE: TRUE

报错：提供的初始场数据维度不对应。

修改：

vim user_nl_clmuse_init_interp=.true.

五：不同的PE设置对FHIST_BGC实验运行速度和效率的影响

实验	节点	运行时长	输出	分辨率
BHIST	9	5:50	三个月，月值	f09_g17
FHIST_BGC	5	4:50	三个月，月值	f09_f09_mg17

初始文件的日期与模式开始运行时间RUN_STARTDATE不一致，会报如下错误：

实验目的：不同的PE设置对FHIST_BGC实验运行速度和效率的影响。

1.1 FHIST_BGC_1 10节点

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_1 &> FHIST_BGC_1.log

1.2 FHIST_BGC_2 20节点

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_2 &> FHIST_BGC_2.log

1.3 FHIST_BGC_3 40节点

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_3 &> FHIST_BGC_3.log

六：Data flow during a model run 模型运行过程中的数据流

./xmlquery RUNDIR./xmlquery DOUT_S,DOUT_S_ROOT

[sysu_syli_3@ln42%tianhe2-H B1850]$ ./xmlquery RUNDIRRUNDIR: /BIGDATA1/sysu_syli_3/CESM2/scratch/B1850/run[sysu_syli_3@ln42%tianhe2-H B1850]$ ./xmlquery DOUT_S,DOUT_S_ROOT
Results in group run_data_archiveDOUT_S: FALSE
Results in group run_doutDOUT_S_ROOT: /BIGDATA1/sysu_syli_3/CESM2/scratch/B1850/archive

日志文件：log file

所有组件的日志文件都会被复制到$CASEROOT/logs 目录下。
如果模型中止，日志文件不会被复制到$RUNDIR 目录下。

输出数据

情况有三：

1. 默认是放置在$RUNDIR 目录下。
2. 如果开启了短期归档，则放置在$DOUT_S_ROOT 目录下。
3. 如果开启了长期归档，则放置在$DOUT_L_MSROOT 目录下。
   Once a model run has completed successfully, the output data flow will depend on whether or not short-term archiving is enabled (as set by the env_run.xml variable, $DOUT_S). By default, short-term archiving will be done.

No archiving

If no short-term archiving is performed, then all model output data will remain in the run directory, as specified by the env_run.xml variable, $RUNDIR.Furthermore,ifshort?termarchivingisdisabled,thenlong?termarchivingwillnotbeallowed.如果禁用了短期归档，则所有的模型输出数据仍然放在‘$RUNDIR` 目录下。如果禁用了短期归档，长期归档也将被禁用。

Short-term archiving

If short-term archiving is enabled, the component output files will be moved to the short term archiving area on local disk, as specified by $DOUT_S_ROOT. The directory DOUT_S_ROOT is normally set to $EXEROOT/../archive/$CASE. and will contain the following directory structure:

atm/hist/ logs/
cpl/ hist/ logs/
glc/ logs/
ice/ hist/ logs/
lnd/ hist/ logs/
ocn/ hist/ logs/
rest/ yyyy-mm-dd-sssss/....yyyy-mm-dd-sssss/

hist/ contains component history output for the run.

logs/ contains component log files created during the run. In addition to $LOGDIR, log files are also copied to the short-term archiving directory and therefore are available for long-term archiving.

rest/ contains a subset of directories that each contain a consistent set of restart files, initial files and rpointer files. Each sub-directory has a unique name corresponding to the model year, month, day and seconds into the day where the files were created (e.g. 1852-01-01-00000/). The contents of any restart directory can be used to create a branch run or a hybrid run or back up to a previous restart date.

Long-term archiving

For long production runs that generate many giga-bytes of data, you will normally want to move the output data from local disk to a long-term archival location. Long-term archiving can be activated by setting $DOUT_L_MS to TRUE in env_run.xml. By default, the value of this variable is FALSE, and long-term archiving is disabled. If the value is set to TRUE, then the following additional variables are: $DOUT_L_MSROOT, $DOUT_S_ROOT DOUT_S (see variables for output data management ).

As was mentioned above, if long-term archiving is enabled, files will be moved out of $DOUT_S_ROOT to $DOUT_L_ROOT by $CASE.l_archive, which is run as a separate batch job after the successful completion of a model run.

当运行完成之后，短期归档archive目录将被创建，历史输出和日志文件被移到这里。run目录下只保留继续运行所必须的文件。
When complete, a short-term archive directory is created, and history and log files are moved here. Files necessary to continue the run are left in $RUNDIR.

七：自定义CAM的强迫

CAM: How do I customize CAM forcings?

参考：CESM1官网：http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/x2349.html

To set the greenhouse gas forcings, you must first understand the namelist variables associated with them. See CAM Namelist Variables for a complete list of CAM namelist variables.

scenario_ghg

• Controls treatment of prescribed co2, ch4, n2o, cfc11, cfc12 volume mixing ratios. May be set to ‘FIXED’ or ‘RAMPED’ or ‘RAMP_CO2_ONLY’.
• FIXED => volume mixing ratios are fixed and have either default or namelist input values.
• RAMPED => volume mixing ratios are time interpolated from the dataset specified by bndtvghg.
• RAMP_CO2_ONLY => only co2 mixing ratios are ramped at a rate determined by the variables ramp_co2_annual_rate, ramp_co2_cap, and ramp_co2_start_ymd.
• Default: FIXED

bndtvghg

• Full pathname of time-variant boundary dataset for greenhouse gas surface values.

rampyear_ghg

• If scenario_ghg is set to “RAMPED” then the greenhouse gas surface values are interpolated between the annual average values read from the file specified by bndtvghg. In that case, the value of this variable (> 0) fixes the year of the lower bounding value (i.e., the value for calendar day 1.0) used in the interpolation. For example, if rampyear_ghg = 1950, then the GHG surface values will be the result of interpolating between the values for 1950 and 1951 from the dataset. Default: 0

Edit user_nl_cam and add the following lines at the end of the file. The following assumes that “my_inputdata_path” is identical to $DIN_LOC_ROOT.

scenario_ghg = 'RAMPED'   # ghg:greenhouse gases
bndtvghg = 'my_inputdata_path/atm/cam/ggas/ghg_hist_1765-2005_c091218.nc'
rampyear_ghg = 2000   

来源：http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/modelnl/nl_cam.html

./create_newcase --compset FHIST_BGC --res f09_f09_mg17 --case ~/CESM2/scratch/FHIST_BGC_CO2 &> FHIST_BGC_CO2.log
cd ~/CESM2/scratch/FHIST_BGC_CO2./xmlquery NTASKS,NTHRDS,ROOTPE
./xmlchange NTASKS=-5
./xmlchange NTASKS_ESP=5./case.setupvim user_nl_clm
hist_empty_htapes = .true 
hist_avgflag_pertape = 'A'
hist_nhtfrq = -24  
hist_mfilt = 1
hist_fincl1="QOVER","QH2OSFC","H2OSOI","GPP","NPP","NEP","TLAI","FLDS","FSDS","Rnet","QVEGT","QSOIL","QINTR","FSH","TSA","Q2M"vim user_nl_cam
scenario_ghg = 'RAMPED'   
bndtvghg = '/BIGDATA1/sysu_syli_3/CESM2/inputdata/atm/cam/ggas/ghg_hist_1765-2005_c091218.nc'
rampyear_ghg = 1979empty_htapes = .true. 
avgflag_pertape='A'
nhtfrq=-24
mfilt = 5
fincl1='T','PS','FSDS'
fincl2=' '
fincl3=' '
fincl4=' '
fincl5=' '
fincl6=' './xmlchange DOUT_S=TRUE
./xmlchange JOB_WALLCLOCK_TIME=24:00:00   ./xmlquery STOP_OPTION,STOP_N,RUN_TYPE,CONTINUE_RUN,RUN_REFCASE,RUN_REFDATE,RESUBMIT,RUN_STARTDATE,RUN_REFDIR,DOUT_S,DOUT_S_ROOT,JOB_WALLCLOCK_TIME,GET_REFCASE./case.build --skip-provenance-check &> FHIST_BGC_CO2.build.log
./case.submit &> FHIST_BGC_CO2.submit.log

err=ERROR(Build::Namelist::_parse_next): expect a F90 constant for a namelist instead got: .true

ERROR: Command /BIGDATA1/sysu_syli_3/CESM2/my_cesm_sandbox/components/cam/bld/build-namelist -ntasks 120 -csmdata /BIGDATA1/sysu_syli_3/CESM2/inputdata -infile /BIGDATA1/sysu_syli_3/CESM2/scratch/FHIST_BGC_CO2/Buildconf/camconf/namelist -ignore_ic_year -use_case hist_cam6 -inputdata /BIGDATA1/sysu_syli_3/CESM2/scratch/FHIST_BGC_CO2/Buildconf/cam.input_data_list -namelist " &atmexp /" failed rc=255
out=
err=CAM build-namelist - ERROR: When flbc_file is used cannot set scenario_ghg = ‘RAMPED’

第一个实验：修改CAM的默认输出

参考：
https://bb.cgd.ucar.edu/cesm/threads/sc-waccm-how-to-reduce-output-to-a-single-monthy-mean-tape-cam2-h0.1560/

vim user_nl_cam
empty_htapes = .true. 
avgflag_pertape='A'
nhtfrq=-24
mfilt = 5
fincl1='T','PS','FSDS'
fincl2=' '
fincl3=' '
fincl4=' '
fincl5=' '
fincl6=' '

完全关闭大气模块的默认输出

大气模块默认月值输出

第二个实验：Change CO2 concentration

参考：
https://bb.cgd.ucar.edu/cesm/threads/change-co2-concentration-in-cesm2-fhist-experiment.5403/#post-37013

八：大气模块

CESM2 中可以运行的大气模块有：

• CAM: Community Atmosphere Model
• CAM-chem: Community Atmosphere Model with Chemistry
• WACCM: Whole Atmosphere Community Climate Model
• WACCM-X: Whole Atmosphere Community Climate Model with thermosphere and ionosphere extension

CAM compsets include the F, P and Q compsets.

F: CAM standalone runs, using an active Atmosphere and Land with prescribed Sea-Surface Temperatures (SSTs) and sea-ice extent.
P: Parallel offline radiation tool (PORT)
Q: Aquaplanet with either prescribed ocean (QP) or slab ocean(QS)

示例1：2倍的CO2对通量的影响 Using PORT to study flux differences due to 2 x CO2

https://ncar.github.io/CAM/doc/build/html/CAM6.0_users_guide/atmospheric-configurations.html

4.2.2.2.1. Sample the base run
4.2.2.2.2. PORT validation
4.2.2.2.3. Compute forcing due to a change in composition (CO2, as an example)
Create the base sampling case:

% cd cime/scripts
% ./create_newcase --case base_run_case --res f09_f09_mg17 --compset F2000climo
% cd base_run_case
% ./case.setup

Set up the user_nl_cam file for the base run:

! Output the radiation data
rad_data_output=.true.! Specify the radiation data be written to history file number 2 (rad_data will be in files with cam.h1 in their name)
rad_data_histfile_num=2! Write out the instantaneous rad_data and radiation diagnostics
rad_data_avgflag = 'I'
avgflag_pertape = 'A','I'! Make certain the radiation is called every time step
iradlw = 1
iradsw = 1! Include radiation diagnostics
fincl2 = 'FLNT', 'FLNR','FLNS', 'FSNT','FSNR', 'FSNS'! Output frequency
nhtfrq = 0,73! number of time records per individual history file
mfilt = 1,5! double precision output
ndens = 1,1

例子： 修改namelist——使用 CMIP5排放

5.2.1. Modifying Namelist settings: Detailed Example – Using CMIP5 emissions

在user_nl_cam 中更改 ext_frc_specifier 和 srf_frc_specifier选项。

ext_frc_specifier  = 'H2O -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/elev/H2O_emission_CH4_oxidationx2_elev_1850-2100_CCMI_RCP6_0_c160219.nc','SO2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/ccmi_1960-2008/IPCC_emissions_volc_SO2_1850-2100_1.9x2.5_c130426cycle.nc','bc_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_bc_elev_1850-2005_c090804.nc','num_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam4_num_a1_elev_1850-2005_c150205.nc','num_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_num_a2_elev_1850-2005_c090804.nc','num_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam4_num_a4_elev_1850-2005_c150205.nc','pom_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_pom_elev_1850-2005_c130424.nc','so4_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a1_elev_1850-2005_c090804.nc','so4_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a2_elev_1850-2005_c090804.nc'srf_emis_specifier = 'DMS -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/ccmi_1950_2100_rcp6/IPCC_emissions_DMS_surface_1850-2100_1.9x2.5_c130814.nc','SO2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/ccmi_1950_2100_rcp6/IPCC_emissions_SO2_surface_1850-2100_1.9x2.5_c130814.nc','SOAG -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_soag_1.5_surf_1850-2005_c130424.nc','bc_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_bc_surf_1850-2005_c090804.nc','num_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam4_num_a1_surf_1850-2005_c150205.nc','num_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_num_a2_surf_1850-2005_c090804.nc','num_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam4_num_a4_surf_1850-2005_c150205.nc','pom_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_pom_surf_1850-2005_c130424.nc','so4_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a1_surf_1850-2005_c090804.nc','so4_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a2_surf_1850-2005_c090804.nc'

自定义模型输出

7.2. Customizing output History Fields

CAM6 做多支持10个历史文件输出流，用户可通过namelist 参数自定义，常见的namelist参数如下：

• finclX - List the fields to include in the output file #X (X=1-10)

• fexclX - List the fields to exclude from the output file #X (X=1-10)
  The following three namelist variables are arrays up to length 10 which specify characteristics for the output files.

• nhtfrq - Array of write frequencies for each history file series. If nhtfrq(1) = 0, the file will be a monthly average. Only the first file series may be a monthly average. If nhtfrq(i) > 0, frequency is specified as number of timesteps. If nhtfrq(i) < 0, frequency is specified as number of hours.

• ndens - set to 1 to output double precision reals, and 2 to output single precision

• mfilt - the maximum number of times to output into a file for each output stream
  There are also namelist settings which control output in a general way.

• empty_htapes - turn off all default output and only write out the fields explicitly set via fincl settings

• history_YYY - add fields for specific diagnostic purposes to the default output. For the complete listing go to the namelist page and search for namelist variables with the history_ prefix (i.e. history_amwg, history_clubb, etc.)

7.3. General Features of History Files

Each time sample in a history file has an associated timestamp which conforms to the CF metadata conventions. The time unit used in CAM’s output files is “days since reference date” where the reference date is the run start date by default, but can be customized via the ref_ymd and ref_tod namelist variables. The variables relevant to the timestamps are the following (from the output of the NetCDF ncdump utility):
历史文件的每个时间样本都会有一个时间戳与之对应，时间单位是“自参考日期以来的天数”，参考日期默认设置为运行开始日期。当然也可以通过ref_ymd 和ref_tod namelist参数自定义。

7.4. Timestamps and time intervals

历史文件中与每个时间样本相关联的时间戳是模型写入数据到磁盘的时间步长末的时间。
The timestamp associated with each time sample in a history file is the model time at the end of the timestep during which the model writes data to the disk. In the case of instantaneous data the meaning is clear. However when the data is representative of a time interval, the timestamp corresponds to the end of the interval.