oracle 无法 fork 新的进程
今天客户的系统突然遇到一个问题，说不能连接新的进程。
这里做个札记，简单记录下，基本诊断如下：
首先先看下系统层面是否有问题：
#errpt -aj xxxxxx
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Sun Feb 23 08:05:14 BEIS
Sequence Number: 381582
Machine Id: 00CF786F4C00
Node Id: xxxxx
Class: S
Type: PERM
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE ---------------------------这里报错已经很明确了：系统交换空间不足
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
nmon_aix52ml5
USER'S PROCESS ID:
153542
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
204
$
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
接着 ，我们再看下alert.log 如下：
alert.log
Sun Feb 23 09:23:32 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:49 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:49 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:50 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:52 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:53 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:55 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:06 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:06 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:16 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:16 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:27 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:27 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:38 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:38 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:49 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:49 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:59 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:59 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:10 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:10 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:21 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:21 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:32 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:32 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:42 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:42 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:53 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:53 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:26:04 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:26:04 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:26:14 2014
注意这一点 ：
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
这里的depinfo =12 的 os 定义为：
#define ENOMEM 12 /* Not enough space */
3. 这里我们再确认下系统配置情况：
系统内存 ： 8g
交换空间 ：
$ lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk1 rootvg 8192MB 35 yes yes lv
oracle的内存配置为 2.3g
根据mos：OracleDatabase (RDBMS) on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix OperatingSystems Installation and Configuration Requirements Quick Reference (8.0.5 to11.2) (文档 ID 169706.1) 确认，当前的系统参数配置完全恰当
for aix ora9i swap ：1Gbor match RAM
最后，推断为可能是用户在短期内做了某些，大排序的操作等，导致oracle 内存耗尽，进而使用os的swap 空间，最后系统资源不足，oracle 无法fork新的进程。
后来系统突然又正常了，因此也没做什么操作。
总结：
1，根据os配置内存和实际业务情况，来确定 sga的大小，并且要保证os有足够内存使用。
2， 如果os内存过小，可以适当调大 swap空间的大小，可以在一定层面上缓解上述状况。
3， 实在万不得已重启下服务器吧！！！！！
FYI：http://blog.csdn.net/eygle/article/details/84166
aix 环境下不封depinfo = xx 代表的错误号信息 :
http://blog.csdn.net/lixora/article/details/19758549