Summary
- DataPump export and import experience hangs when processing /STATISTICS/MARKER:
Completed 1 MARKER objects in 1774 seconds
- The time is spent on:
insert into "SYS"."IMPDP_STATS" select null, type, version, flags, c1, c2, c3, c4, c5, c6, n1, n2, n3, n4, n5, n6, n7,
n8, n9, n10, n11, n12, n13, d1, t1, r1, r2, r3, ch1, cl1 from sys."_user_stat" st where type in ('T','C','E','P','M','U','G','L')
and (type not in ('t','c','i') or :1 is null or :2 < t1) and c5 = :3 and c1 = :4
which has an execution plan where a lot of time is spent in 2 row source INDEX RANGE scans on I_OBJ2:
0 0 0 NESTED LOOPS (cr=1994 pr=0 pw=0 time=45315 us cost=12 size=100 card=1)
0 0 0 NESTED LOOPS (cr=1994 pr=0 pw=0 time=45312 us cost=12 size=100 card=1)
0 0 0 NESTED LOOPS (cr=1994 pr=0 pw=0 time=45309 us cost=11 size=74 card=1)
0 0 0 NESTED LOOPS (cr=1994 pr=0 pw=0 time=45306 us cost=9 size=88 card=2)
0 0 0 NESTED LOOPS (cr=1994 pr=0 pw=0 time=45303 us cost=8 size=64 card=2)
1 1 1 TABLE ACCESS BY INDEX ROWID USER$ (cr=2 pr=0 pw=0 time=16 us cost=1 size=17 card=1)
1 1 1 INDEX UNIQUE SCAN I_USER1 (cr=1 pr=0 pw=0 time=7 us cost=1 size=0 card=1)(object id 10068169)
0 0 0 INDEX RANGE SCAN I_OBJ2 (cr=1992 pr=0 pw=0 time=45279 us cost=7 size=30 card=2)(object id 10068133) <<<<<<
0 0 0 TABLE ACCESS BY INDEX ROWID TABSUBPART$ (cr=0 pr=0 pw=0 time=0 us cost=1 size=12 card=1)
53 39819 184103 NESTED LOOPS (cr=1994 pr=0 pw=0 time=34223 us cost=8 size=338576 card=12092)
1 1 1 TABLE ACCESS BY INDEX ROWID USER$ (cr=2 pr=0 pw=0 time=17 us cost=1 size=17 card=1)
1 1 1 INDEX UNIQUE SCAN I_USER1 (cr=1 pr=0 pw=0 time=8 us cost=1 size=0 card=1)(object id 10068169)
53 39819 184103 INDEX RANGE SCAN I_OBJ2 (cr=1992 pr=0 pw=0 time=64681 us cost=7 size=133012 card=12092)(object id 10068133) <<<<
- Patch for the unpublished BUG 21644640 - 12C (12.1.0.2) EXPDP HANGS ON INSERT INTO TABLE IMPDP_STATS, has been applied, but the issue is still there.
Solution
Use any of below alternatives until the bugs are fixed:
- 1/ Create the following index:
create index SYS.IMPDP_STATS_1 ON SYS.IMPDP_STATS (c5,type,c1,c2,c3,c4,statid,version);
and re-run the DataPump job
这个insert的表建index 什么意思？ 不应该是sys."_user_stat" ？
-- Create table
create global temporary table SYS.IMPDP_STATS
(
statid VARCHAR2(128),
type CHAR(1),
version NUMBER,
flags NUMBER,
c1 VARCHAR2(128),
c2 VARCHAR2(128),
c3 VARCHAR2(128),
c4 VARCHAR2(128),
c5 VARCHAR2(128),
c6 VARCHAR2(128),
n1 NUMBER,
n2 NUMBER,
n3 NUMBER,
n4 NUMBER,
n5 NUMBER,
n6 NUMBER,
n7 NUMBER,
n8 NUMBER,
n9 NUMBER,
n10 NUMBER,
n11 NUMBER,
n12 NUMBER,
n13 NUMBER,
d1 DATE,
t1 TIMESTAMP(6) WITH TIME ZONE,
r1 RAW(1000),
r2 RAW(1000),
r3 RAW(1000),
ch1 VARCHAR2(1000),
cl1 CLOB,
bl1 BLOB
)
on commit preserve rows;
-- Create/Recreate indexes
create index SYS.IMPDP_STATS on SYS.IMPDP_STATS (STATID, TYPE, C5, C1, C2, C3, C4, VERSION);
-- Grant/Revoke object privileges
grant select, insert, update, delete on SYS.IMPDP_STATS to PUBLIC;
or
- 2/ Run the DataPump job with exclude=statistics.
Attachments :
Cause
The issue is still investigated by Development in Query Optimizer bugs:
for 12.1:
-
<Bug 23564853> - DATAPUMP HANG AT STATISTICS MARKER - This was closed as duplicate of the following:
-
unpublished Bug 32614157 - INLINST OPTIMIZATION NOT USED WHEN PRIOR COLUMN HAS AN "IS NULL" PREDICATE - This fix will be included in version 23.1
-
unpublished Bug 24961125 - DATAPUMP IMPORT FROM TABLES WITH MANY PARTITIONS IS SLOW - This was determined not feasible to fix by development
for 12.2:
- unpublished Bug 25482506 - DATAPUMP EXPORT SLOW WHEN UNLOADING STATISTICS/MARKER- This was determined not feasible to fix by development
Summary
After upgrading to 19c, data pump jobs take long time during STATISTICS/MARKER.
SQL trace for data pump worker process shows following SQL with high elapsed time:
insert /*+ no_gather_optimizer_statistics */ into "SYS"."IMPDP_STATS" (
statid,type,version,flags,c1,c2,c3,c4, c5,c6,n1,n2,n3,n4,n5,n6,n7,n8,n9,n10,
n11,n12,n13,d1,t1,r1,r2,r3,ch1,cl1 ,bl1 ) select null ,type,version,flags,
c1,c2,c3,c4,c5,c6,n1,n2,n3,n4,n5,n6,n7,n8,n9,n10,n11,n12,n13,d1,t1,r1,r2,r3,ch1, cl1 ,bl1
from sys."_user_stat" st
where (type in ('T','C','E','P','M','U','G','L')
or (type in ('c', 't')
and (t1 in (:1, :2)
or (:3 is not null and t1 > :4 and t1 < :5))))
and c5 = :6
and c1 = :7
and min_sc <= 2
and 2 < max_sc
call count cpu elapsed disk query current rows
Parse 33202 2.19 2.20 0 0 0 0
Execute 33202 2082.58 2244.31 98877 228298446 32928257 6146328 变量次数多
Fetch 0 0.00 0.00 0 0 0 0
total 66404 2084.78 2246.51 98877 228298446 32928257 6146328
Solution
Apply <Patch 34000704> if available for your version and platform.
To check for conflicting patches, please use the My Oracle Support (MOS) Patch Planner Tool.
Reference:
How to Use MOS Patch Planner to Check and Request the Conflict Patches? (<Note 1317012.1>)
If no patch exists for your platform and version, please contact Oracle Support to request a fix.
Attachments :
Cause
Upgrade from 11.2.0.4.0 to 19.12.0.0.0
Data pump export can take a long time on databases with thousands of table and index partitions. This is seen in the data pump log as time taken for the STATISTICS/MARKER object type.
This issue was investigated in following internal bugs.
Bug 34622579 - METADATA EXPDP HANGS ON STATISTICS INSERT INTO IMPDP_STATS
which was closed as duplicate of internal bug:
Bug 34000704 - EXPDP HANGS ON INSERT INTO SYS"."IMPDP_STATS
Summary
Experiencing slow DataPump Export starting with 12c when exporting statistics and DW00 process is doing 'insert into "SYS"."IMPDP_STATS" ...'.
Tkprof output of the DW00 process shows very long execution plan for insert into "SYS"."IMPDP_STATS" statement:
insert into "SYS"."IMPDP_STATS" select null, type, version, flags,c1, c2, c3, c4, c5, c6, n1, n2, n3, n4, n5, n6, n7, n8, n9, n10, n11, n12, n13, d1, t1, r1, r2, r3, ch1, cl1 from sys."_user_stat" st
where type in ('T','C','E','P','M','U','G','L') and (type not in ('t','c','i') or :1 is null or :2 < t1) and c5 = :3 and c1 = :4
call count cpu elapsed disk query current rows
Parse 66496 9.28 9.59 0 0 18 0
Execute 66496 2632.34 10644.74 70604 25026581 13280088 3214726
Fetch 0 0.00 0.00 0 0 0 0
total 132992 2641.62 10654.33 70604 25026581 13280106 3214726
Rows1st RowsAvg RowsMax Row Source Operation
0 0 0 LOAD TABLE CONVENTIONAL (cr=7547 pr=0 pw=0 time=2211494964 us)
138 138 138 HASH JOIN (cr=7544 pr=0 pw=0 time=74760 us cos
48 48 48 VIEW VW_NSO_2 (cr=3 pr=0 pw=0 time=209 us cos
48 48 48 SORT UNIQUE (cr=3 pr=0 pw=0 time=114 us cost=
48 48 48 TABLE ACCESS FULL SPD_SCRATCH_TAB (cr=3 pr=0
43027 43027 43027 VIEW (cr=7541 pr=0 pw=0 time=2298269903 us co
43027 43027 43027 UNION-ALL (cr=7541 pr=0 pw=0 time=2298242658
0 0 0 TABLE ACCESS CLUSTER TAB$ (cr=0 pr=0 pw=0 ti
0 0 0 INDEX UNIQUE SCAN I_OBJ# (cr=0 pr=0 pw=0 ti
:
:
15426 15426 15426 NESTED LOOPS(cr=6175 pr=0 pw=0 time=1728620348
3010261 3010261 3010261 HASH JOIN (cr=6175 pr=0 pw=0 time=5086802
6262 6262 6262 TABLE ACCESS FULL OPT_DIRECTIVE$ (cr=61 p
3010261 3010261 3010261 HASH JOIN (cr=6114 pr=0 pw=0 time=253386
27601 27601 27601 TABLE ACCESS FULL OPT_FINDING_OBJ$ (cr=2
404035 404035 404035 HASH JOIN (cr=5894 pr=0 pw=0 time=11517
107 107 107 TABLE ACCESS FULL USER$ (cr=9 pr=0 pw=0
404035 404035 404035 HASH JOIN (cr=5885 pr=0 pw=0 time=9052
95174 95174 95174 VIEW (cr=1067 pr=0 pw=0 time=63858 us
95174 95174 95174 UNION-ALL (cr=1067 pr=0 pw=0 time=46
92769 92769 92769 INDEX FAST FULL SCAN I_OBJ5 (cr=1067
2405 2405 2405 VIEW GV$FIXED_TABLE (cr=0 pr=0 pw=0
2405 2405 2405 UNION-ALL (cr=0 pr=0 pw=0 time=177
1107 1107 1107 FIXED TABLE FULL X$KQFTA (cr=0 pr=
1261 1261 1261 FIXED TABLE FULL X$KQFVI (cr=0 pr=
37 37 37 FIXED TABLE FULL X$KQFDT (cr=0 pr=
404035 404035 404035 VIEW (cr=4818 pr=0 pw=0 time=537971 u
404035 404035 404035 UNION-ALL (cr=4818 pr=0 pw=0 time=44
385890 385890 385890 TABLE ACCESS FULL COL$ (cr=4818 pr=0
18145 18145 18145 FIXED TABLE FULL X$KQFCO (cr=0 pr=0
15426 15426 15426 COLLECTION ITERATOR PICKLER FETCHGET_VEC_SET_IDS (cr=0 pr=0 pw=0 time=2751892245 us cost=27 size=54 card=27)
0 0 0 FILTER (cr=0 pr=0 pw=0 time=2 us)
0 0 0 TABLE ACCESS FULL OPTSTAT_HIST_CONTROL$ (c
1 1 1 FIXED TABLE FULL X$KZSPR (cr=0 pr=0 pw=0 tim
0 0 0 FIXED TABLE FULL X$KZSPR (cr=0 pr=0 pw=0 tim
0 0 0 FIXED TABLE FULL X$KZSPR (cr=0 pr=0 pw=0 tim
Solution
To avoid Bug 21644640, implement any of the below alternative solutions:
- Upgrade to 12.2 when it will become available
- OR -
- Download and apply interim Patch 21644640, if available for your platform and RDBMS release. To check for conflicting patches, please use the MOS Patch Planner Tool. If no patch is available, file a Service Request through My Oracle Support for your specific Oracle version and platform
- OR -
- Workaround: Use exclude=statistics
Cause
The DataPump performance issue is due to 0 cardinality for OBJ/USER indexes after instance restart.
In some cases insert into table impdp_stats is slow due to a bad optimizer plan. The poor execution plan is chosen due to an issue in stats load for bootstrap indexes.
This optimizer issue has been investigated by Development in Bug 21644640 - 12C (12.1.0.2) EXPDP HANGS ON INSERT INTO TABLE IMPDP_STATS, fixed in 12.2.