Golden Gate: ERROR OGG-01224 Address already in use.

PROBLEM:

ERROR OGG-01224 Address already in use.

 

GGSCI (dc174.ora.com) 34> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED

EXTRACT     STOPPED     EXTORA1     00:00:00      00:01:08

EXTRACT     STOPPED     PUMPORA1    00:00:00      00:01:06

GGSCI (dc174.ora.com) 35> start mgr

Manager started.

GGSCI (dc174.ora.com) 37> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED

EXTRACT     STOPPED     EXTORA1     00:00:00      00:01:16

EXTRACT     STOPPED     PUMPORA1    00:00:00      00:01:14

GGSCI (dc174.ora.com) 38> view report mgr

…

…

Operating System Version:

Linux

Version #1 SMP Mon Sep 30 16:34:30 PDT 2013, Release 2.6.18-371.el5

Node: dc174.ora.com

Machine: x86_64

                         soft limit   hard limit

Address Space Size   :    unlimited    unlimited

Heap Size            :    unlimited    unlimited

File Size            :    unlimited    unlimited

CPU Time             :    unlimited    unlimited

Process id: 13933

Parameters...

PORT 7809

SYSLOG NONE

***********************************************************************

**                     Run Time Messages                             **

***********************************************************************

Source Context :

  SourceModule            : [mgr.main]

  SourceID                : [/scratch/aime1/adestore/views/aime1_adc41502

56/oggcore/OpenSys/src/app/mgr/mgr.c]

  SourceFunction          : [init_functions]

  SourceLine              : [3264]

  ThreadBacktrace         : [8] elements

                          : [/u01/app/gg_home/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x2b7eee20106e]]

                          : [/u01/app/gg_home/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x2b7eee1fd44c]]

                          : [/u01/app/gg_home/libgglog.so(_MSG_ERR_TCP_GENERIC(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x2b7eee1e251f]]

                          : [./mgr(init_functions(int, char**)+0x7f5) [0x448235]]

                          : [./mgr(main_loop(int, char**)+0x4c) [0x44e13c]]

                          : [./mgr(main+0xdc) [0x44e8bc]]

                          : [/lib64/libc.so.6(__libc_start_main+0xf4) [0x

3f5a01d9c4]]

                          : [./mgr(__gxx_personality_v0+0x14a) [0x43a49a]]

2015-01-09 15:20:57  ERROR   OGG-01224  Address already in use.

2015-01-09 15:20:57  ERROR   OGG-01668  PROCESS ABENDING.

PROBLEM:

mgr cannot start. "ERROR   OGG-01224  Address already in use and ERROR   OGG-01668  PROCESS ABENDING." appear in mgr.rpt

CAUSE:

Previous mgr process is still runing, which used port 7809 according to parameter file.

SOLUTION:

as root or Goldengate owner:

# netstat -nap | grep 7809

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)

tcp        0      0 :::7809                     :::*

LISTEN      29478/./mgr

# kill -9 29478

Restart MGR process.

GGSCI (dc174.ora.com) 1> start mgr

MGR is already running.

GGSCI (dc174.ora.com) 2> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     STOPPED     EXTORA1     00:00:00      00:21:32

EXTRACT     STOPPED     PUMPORA1    00:00:00      00:21:30

opiodr aborting process unknown ospid as a result of ora-609

opiodr aborting process unknown ospid as a result of ora-609

TNS-12560: TNS:protocol adapter error

ORA-609 : opiodr aborting process unknown ospid (2230_87567)

·         Details on the ORA-609 error, see MOSC note 1121357.1.

As a general error, the ORA-609 error indicates that a client connection failed to complete.  This can be an ORA-609 from an abort or killing an Oracle session.  

To diagnose any error, you start by using the OERR UTILITY to display the ORA-609 error:

Example :

bash-3.2$ oerr ora 609

00609, 00000, "could not attach to incoming connection"

// *Cause:  Oracle process could not answer incoming connection

// *Action: If the situation described in the next error on the stack

//         can be corrected, do so; otherwise contact Oracle Support.

The ORA-609 error is thrown when a client connection of any kind failed to complete or aborted the connection process before the server process was completely spawned. Beginning with 10gR2, a default value for inbound connect timeout has been set at 60 seconds.  This is also triggered when a DB session is killed/aborted manually from the OS prompt.

Solution:

Increase the values for INBOUND_CONNECT_TIMEOUT at both listener and server side sqlnet.ora file as a preventive measure.

If the problem is due to connection timeouts, and increase in the following parameters should eliminate or reduce the occurrence of the ORA-609 error. 

ü  Sqlnet.ora: SQLNET.INBOUND_CONNECT_TIMEOUT=180

ü  Listener.ora: INBOUND_CONNECT_TIMEOUT_listener_name=120

ORA-12560: TNS: protocol adapter error

Cause: A generic protocol adapter error occurred.

Action: Check addresses used for proper protocol specification. Before reporting this error, look at the error stack and check for lower level transport errors. For further details, turn on tracing and re-execute the operation. Turn off tracing when the operation is complete.

Expected Fix :

1. Check environment variable like $ORACLE_HOME, $PATH, $ORACLE_BASE
2. TNS_ADMIN is set properly
3. Verify firewall that not blocking ports
4. Check permission

OCSSD starts if voting disk & OCR resides in ASM Diskgroups?

Voting & OCR Disk resides in ASM Diskgroups, but as per startup sequence OCSSD starts first before than ASM, how is it possible?

How does OCSSD starts if voting disk & OCR resides in ASM Diskgroups?

You might wonder how CSSD, which is required to start the clustered ASM instance, can be started if voting disks are stored in ASM? 

This sounds like a chicken-and-egg problem: without access to the voting disks there is no CSS, hence the node cannot join the cluster. But without being part of the cluster, CSSD cannot start the ASM instance. 

To solve this problem the ASM disk headers have new metadata in 11gR2 (11.2.x.x) you can use kfed to read the header of an ASM disk containing a voting disk. 

The kfdhdb.vfstart and kfdhdb.vfend fields tell CSS where to find the voting file. This does not require the ASM instance to be up. Once the voting disks are located, CSS can access them and joins the cluster.

Use kfed to read ASM disk header block.

# displays online help for the utility

$ kfed -help

# reads the disk header to stdout

$ kfed op=read dev=/dev/mapper/postrf741_11p1

# reads the specified AU and block into file /tmp/a

$ kfed op=read dev=/dev/mapper/postrf741_11p1 aunum=3 blknum=3 text=/tmp/a

# writes from /tmp/a into the specified AU and block
#block checksum is computed and written together with data

$ kfed op=write dev=/dev/mapper/postrf741_11p1 aunum=3 blknum=3 text=/tmp/a

Oracle Kernel RAC: ON and OFF

Oracle Kernel needs to switch with ORACLE RAC ON / OFF when you convert from Standalone to RAC or vice versa.

In some cases, you may want to enable or disable the RAC options for testing purposes –

Disabling and enabling RAC options are available only for UNIX/LINUX platforms.

I haven’t checked on Windows RAC Instances.

Use the following steps to disable RAC (known as RAC OFF):

1.         Log in as the Oracle software owner in my case (oracle) (which is typically the UNIX account oracle) in all nodes.

2.         Shutdown normally all the instances on each and every node.

3.     Go to $ORACLE_HOME/rdbms/lib:
$ cd $ORACLE_HOME/rdbms/lib

4.         Execute below make command to relink the Oracle binaries without the RAC option:

make -f ins_rdbms.mk rac_off

rebuild oracle

This normally runs for few minutes and should not pose any errors.
(if any error fixed it or immediate contact with oracle support)

5.         Now relink the Oracle binaries:

cd $ORACLE_HOME/bin
#make -f ins_rdbms.mk oracle

relink oracle

6.         Now the Oracle binaries are relinked with the RAC OFF option.
You may have to edit the parameter file init.ora or SPFILE parameters accordingly (Cluster_Database, Undo, Threads) etc.

Use the following steps to enable RAC (known as RAC ON):

1.                   Log in as the Oracle software owner in my case (oracle) (which is typically the UNIX account oracle) in all nodes.

2.                   Shutdown normally all the instances on each and every node.

3.               Go to $ORACLE_HOME/rdbms/lib:
$ cd $ORACLE_HOME/rdbms/lib

4.                   Execute the following make command to relink the Oracle binaries without the RAC option:

make -f ins_rdbms.mk rac_on

rebuild oracle

This normally runs for a few minutes and should not pose any errors.
(Get in touch with Oracle Support in case of any error)

5.                   Now relink the Oracle binaries:

cd $ORACLE_HOME/bin

#make -f ins_rdbms.mk oracle

relink oracle

Now the Oracle binaries are relinked with the RAC ON option. You may need to edit the parameter file init.ora or SPFILE parameters accordingly like (Cluster_Database,Undo,Threads…etc).

Manual make commands

While the Oracle relink command will automatically run the make commands, you still may run them menually:

For executables: oracle, exp, imp, sqlldr, tkprof, mig, dbv, orapwd, rman, svrmgrl, ogms and ogmsctl

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk install

For sqlplus

cd $ORACLE_HOME/sqlplus/lib
make -f ins_sqlplus.mk install

For executables: dbsnmp, oemevent, oratclsh

cd $ORACLE_HOME/network/lib
make -f ins_oemagent.mk install

For executables: names, namesctl

cd $ORACLE_HOME/network/lib
make -f ins_names.mk install

For executables: osslogin, trcasst, trcroute, onrsd, tnsping

cd $ORACLE_HOME/network/lib
make -f ins_net_client.mk install

For executables: tnslsnr, lsnrctl

cd $ORACLE_HOME/network/lib
make -f ins_net_server.mk install

For more details on using the relink command, see MOSC Note: 131321.1 titled "Using Oracle relink on UNIX".

GoldenGate - Error ORA-20782 while deleting Golden Gate User

Error ORA-20782 while deleting Golden Gate User

While dropping GoldenGate replicated user, I got an error.  Details are given below.

SQL> select distinct owner

     from dba_segments

     where owner in (select username

                from dba_users

                where default_tablespace not in ('SYSTEM','SYSAUX')

                ) ;

OWNER

------------------------------

GGTEST

OGGSOURCE

OGGTARGET

…

…

SQL> drop user GGTEST cascade;

drop user GGTEST cascade

*

ERROR at line 1:

ORA-00604: error occurred at recursive SQL level 2

ORA-20782: GoldenGate DDL Replication Error: Code :ORA-20782: Cannot DROP

object used in GoldenGate replication while trigger is enabled. Consult

GoldenGate documentation and/or call GoldenGate Technical Support if you wish to do so., error stack: ORA-06512: at line 231

ORA-06512: at line 957

Solution

Since I had run the ddl_setup.sql which created the DDL trigger for Golden Gate user schema.

So before deleting the Golden Gate replicated user, we need to drop the Golden Gate DDL trigger first.

SQL> SELECT * FROM DBA_TRIGGERS

WHERE OWNER='GGTEST';

no rows selected

SQL>  SELECT A.OBJ#, A.SYS_EVTS, B.NAME

FROM TRIGGER$ A,OBJ$ B

WHERE A.SYS_EVTS > 0

AND A.OBJ#=B.OBJ#

AND BASEOBJECT = 0;

      OBJ#   SYS_EVTS NAME

---------- ---------- ------------------------------

     11990     524256 LOGMNRGGC_TRIGGER

     13175       4096 AW_TRUNC_TRG

     13177       8192 AW_REN_TRG

     13179        128 AW_DROP_TRG

     13885       8416 NO_VM_DDL

     13886        128 NO_VM_DROP_A

     55420         64 CDC_ALTER_CTABLE_BEFORE

     55421         32 CDC_CREATE_CTABLE_AFTER

     55422         32 CDC_CREATE_CTABLE_BEFORE

     55423        128 CDC_DROP_CTABLE_BEFORE

     56322         96 EXPFIL_RESTRICT_TYPEEVOLVE

     56323       8256 EXPFIL_ALTEREXPTAB_MAINT

     57549       4224 XDB_PI_TRIG

     56320        128 EXPFIL_DROPOBJ_MAINT

     56321        128 EXPFIL_DROPUSR_MAINT

     58846       4096 RLMGR_TRUNCATE_MAINT

     63703        128 SDO_DROP_USER

     63971         32 SDO_ST_SYN_CREATE

     63807        128 SDO_TOPO_DROP_FTBL

     68065     524256 SDO_GEOR_BDDL_TRIGGER

     68066     524256 SDO_GEOR_ADDL_TRIGGER

     68143        128 SDO_NETWORK_DROP_USER

     71613          1 MGMT_STARTUP

     76915     524256 GGS_DDL_TRIGGER_BEFORE

24 rows selected.

SQL> drop trigger GGS_DDL_TRIGGER_BEFORE;

Trigger dropped.

SQL> drop user GGTEST cascade;

User dropped.

GoldenGate - Overview

GoldenGate enables us to extract and replicate data across a variety of topologies as shown the diagram below as well as the exchange and manipulation of data at the transactional level between a variety of database platforms like Oracle, MySQL, SQL Server, Teradata, UDB, DB2 etc.

It can support a number of different business requirements like:

§  Business Continuity and High Availability

§  Data migrations and upgrades

§  Decision Support Systems and Data Warehousing

§  Data integration and consolidation

Manager

The Manager process must be running on both the source as well as target systems before the Extract or Replicat process can be started and performs a number of functions including monitoring and starting other GoldenGate processes, managing the trail files and also reporting.

Extract

The Extract process runs on the source system and is the data capture mechanism of GoldenGate. It can be configured both for initial loading of the source data as well as to synchronize the changed data on the source with the target. This can be configured to also propagate any DDL changes on those databases where DDL change support is available.

Replicat

The Replicat process runs on the target system and reads transactional data changes as well as DDL changes and replicates then to the target database. Like the Extract process, the Replicat process can also be configured for Initial Load as well as Change Synchronization.

Collector

The Collector is a background process which runs on the target system and is started automatically by the Manager (Dynamic Collector) or it can be configured to stsrt manually (Static Collector). It receives extracted data changes that are sent via TCP/IP and writes then to the trail files from where they are processed by the Replicat process.

Trails

Trails are series of files that GoldenGate temporarily stores on disks and these files are written to and read from by the Extract and Replicat processes as the case may be. Depending on the configuration chosen, these trail files can exist on the source as well as on the target systems. If it exists on the local system, it will be known an Extract Trail or as an Remote Trail if it exists on the target system.

Data Pumps

Data Pumps are secondary extract mechanisms which exist in the source configuration. This is optional component and if Data Pump is not used then Extract sends data via TCP/IP to the remote trail on the target. When Data Pump is configured, the Primary Extract process will write to the Local Trail and then this trail is read by the Data Pump and data is sent over the network to Remote Trails on the target system.

In the absence of Data Pump, the data that the Extract process extracts resides in memory alone and there is no storage of this data anywhere on the source system. In case of network of target failures, there could be cases where the primary extract process can abort or abend. Data Pump can also be useful in those cases where we are doing complex filtering and transformation of data as well as when we are consolidating data from many sources to a central target.

Data source

When processing transactional data changes, the Extract process can obtain data directly from the database transaction logs (Oracle, DB2, SQL Server, MySQL etc) or from a GoldenGate Vendor Access Module (VAM) where the database vendor (for example Teradata) will provide the required components that will be used by Extract to extract the data changes.

Groups

To differentiate between the number of different Extract and Replicat groups which can potentially co-exist on a system, we can define processing groups. For instance, if we want to replicate different sets of data in parallel, we can create two Replicat groups.

A processing group consists of a process which could be either a Extract or Replicat process, a corresponding parameter file, checkpoint file or checkpoint table (for Replicat) and other files which could be associated with the process.