Home

Expert Oracle GoldenGate

1. 222 www it ebooks info Trall File PUTINO 68 222 Trail Files that Purge 001 223 Oracle GoldenGate Error Log Analysis eese eren 223 Understanding the Oracle GoldenGate Discard File 223 Discard File Not Created eese tenerte 224 Discard File Is Too Big eeeeeeeeeenenene tentent nnne nnne nnne nnne 224 Can t Open the Discard File teta asa 225 Using Trace Commands with Oracle GoldenGate sss 225 Oracle GoldenGate Process Tracing with TLTRACE eene 225 Using TRACE Parameters with Oracle GoldenGate enne 226 Troubleshooting Case Study with Oracle GoldenGate ee 226 Oracle GoldenGate Configuration Issues eene
2. 281 sea onini c 282 Creating a Dedicated Installation Directory eeeeeeeee rennen nnne nennen nnn nnn nunt 282 Using a Checkpoint Table eene tnter tentent tnnt tn tnnente tonis 282 Verifying the Character Sets eese essen ee rennen tnnt nn trata trata trata iata inia sn anas 283 Developing Naming Standards esee esee nene nere tn nnne natnra tnra irata trata oa 283 Using a Data Pump eene tentent tnnt tentent 283 Management and Monitoring 1 eeeeeseeseee ener enne 284 Using GGSCI Command Shortcuts eese tentent tnt tnnt 284 anu mee T 285 Generating Interim Statistics sese tetetnte 286 Using Discard 286 Reporting Regularly on Process 286 Purging Old Trail Files Regularly 1eeeeeeeeere rennen nnne nnne nnn nnn nnne nnn nnns 287 Automatically Starting 55 6 anaana
3. 171 GoldenGate Veridata 5 172 GoldenGate Veridata Web esses eeeeeeenee nennen tnntnntntn tnn tns tantas tta tasses tastes 173 GoldenGate Veridata Repository sessi ieiee isses sene nenternn tnn tnntnntnnta tatnen tentant 173 GoldenGate Veridata Agent Java and C Code eere eren ennt nnnna nnns 173 GoldenGate Veridata CLI 173 How Veridata Comparison Works eene 173 How Veridata Dan Help 174 www it ebooks info Setting Up the Veridata 175 Creating Database Connections 175 Setting Up Tables and Data Scripts eeeeeeeeeeeeeeeeee ener 176 177 Creating Compare Pairs nennen tnnt tnnt nte tonnes tette 177 178 Creating Profile
4. 179 Running the Veridata Jobir 179 Improving Performance and Reducing Overhead esses 180 Excluding Columns eeeeeeee eene tnnt tnter tnnt tente 180 Tuning Profiles Settings s seeeeseeeeeeeeee eere eee rennen nennen nnne 180 Disabling the Confirm Out of Sync Step sessi rennen ernannt natnra tano 180 Increasing the Number of Threads 180 UErP 180 Right Trim on Character Fields 181 Comparing Incremental Data for Large Tables eeeeeeeeee nere nennen nennen nnns 181 Comparing GoldenGate Real Time Replication Data 182 Comparing Different Column Types and Compare Formats 182 Using Performance Statistics ainscnasnaiacuer ici mieczenitiriiviziriininititinitixitin uni n vas 183 Using Vericom Command uie 185 Setting Up Role Based Securily ceres
5. 1 1 cere re ere aaa aaan 206 Extracting TranlogoptlOFS s a ocean rtr haie enn 206 Generating Definition Files 207 Finding Parameters or Attributes in Director rennes 209 Advanced MAPPING 25 5222 2 2 27 165 ntur reor nca terraa neuen yrs ce Scy aca tere Fa Ca 210 con ROCHE ORDRE 213 ir H 215 Chapter 11 Troubleshooting Oracle GoldenGate 217 Common Problems and 5 0 217 Oracle GoldenGate Process iki kernel Rua ats 218 Oracle GoldenGate Extract Process Failures eese eenennennn tentant 219 Oracle GoldenGate Process Failures Without Report Diagnostics sess 221 Oracle GoldenGate Trail File Problems nennen 221 Trail Files that Don t Empty sse enenatis tates 221 Trail Files that Don t Roll
6. eese 134 Implementing Parallel Extracts and Replicats with Table Filtering 135 Implementing Parallel Extracts and Replicats Using Key Ranges eene 141 USING BATGHSQL 146 Using GROUP TRANS OPS ct 149 IE Sore Ecran Aaa 149 Tuning the Network eese tnter tnnt tnnt nte tinte tinte tente tinte tinte tns 150 Tuning the RMTHOST 151 Tuning the Database ttai tus anos ion nha ta ron gota tna ba po dE 152 SUMMA M 152 www it ebooks info CONTENTS Chapter 8 Monitoring Oracle GoldenGate 153 Designing a Monitoring Strategy eere rrr eer eerenne 153 Why Monitoring the Extract Is Important esses eeeee rennen rennen nnn nnne nennen nnns 155 Getting the Maximum Threshold tenete 155 Which Processes to Monitor in the GoldenGate Environment
7. nennen enne nennen nnne nnn nnn nennen 49 esee ener nennen nnns sane soa sana so asm us 49 Prerequisites for Setting Up 49 Requirements 50 One Way Replication 1 52 Basic Replication Steps ER 52 www it ebooks info Ri aliia etf ttt 54 Verifying Database Level Supplemental Logging eene 54 Enabling Database Level Supplemental Logging esent 54 Enabling Table Level Supplemental Logging sesenta 55 Disabling Triggers and Cascade Delete Constraints eene 57 Verifying the Manager Status eese eene ene nn enn tn tnnt tn 58 Configuring the Local Extract eese esee eeeee nenne rennen rnnt 59 prinia 1 63 Starting and Stopping the Extract ssscsssssssssssscsesssssssessssssessssesscsessssesesseseesssaesscaeseeaeserseseesesa
8. 296 Accessing the Logdump Utility esssseseeseeeeeeeeeenene nennen 296 Getting Help with Logdump Syntax eres eeeeeeee nennen rennen anna nannte trataron tnos 296 299 Opening GoldenGate Trail Files with Logdump eese tnnt 299 91 www it ebooks info About the Authors Ben Prusinski is an Oracle Certified Professional OCP and database architect with more than 14 years of experience with Oracle data warehouse and Oracle E Business applications As a corporate database consultant Ben has provided services to dozens of Fortune 500 clients He is an internationally recognized expert in Oracle high availability performance tuning database security and ERP implementations As a top Oracle expert Ben received the prestigious Oracle ACE award in 2009 in recognition of his contributions to the Oracle community As an Oracle GoldenGate Certified Associate Ben is a popular speaker at major conferences such as Oracle OpenWorld Chilean Oracle User Group CLOUG in Latin America the Independent Oracle Users Group IOUG and Oracle Collaborate OAUG Oracle Applications User Group In addition to Oracle consulting for clients Ben regularly updates his Oracle blog at http oracle magician blogspot com wi
9. 155 Monitoring All Running PrOCeSSes 1 eneeeeeeeere ne enn rne nntn nnne 156 Monitoring the Detail Extract 156 Checking the Current Oracle Online Redo Log 157 Checking the Archived Log 1 4 1 4 1 creer inaina daiaire aiani nne re ana kaadaa daana asas masa an ama sa amano 157 Monitoring the GoldenGate Rate and Redo Log Consumption Rate sess 158 Monitoring Lags for Each Group eeeeseeeeee eere ener eren nenne nnn 159 Viewing Event and Error Logs eeeeeeereeee rene nnne 162 Automating Monitoring 1 eesessee sees re eren neret nnne nnns nnn 162 Checking GoldenGate Process SCTiIPts s ssssssssessssessessssessssessssessessessessseesesseessesssssneseeseeassseens 162 Monitoring 0 163 Checking Memory and CPU Scripts 168 IIPRD 168 MILI m 1 169 Chapter 9 Oracle GoldenGate Veridata nne 171 Veridata
10. eene 12 Reviewing the Install Instructions rci cniin 14 Installing Goldengate e 16 General System Requirements eese seee ness tenete rte tnn tentes tanta nenne 16 Memory Requirements eene tenerte tete tentent tente tatnen toten tans 16 Disk Space Requiremetits nane nra urn anra 17 Network Requirements eene rennen nnne tete 17 www it ebooks info Operating System 5 18 Requirements for Microsoft Windows Clustered Environments eene 18 Installing Goldengate on Windows cccssssssessesesessssesssseesessesessssessssessesecsesassesaeseeesaseesaesesaesseeasansess 18 Installing GoldenGate 11g on Linux and UNIX eeeeereee rene nennen nnnnnnnnnnnn 20 Environment Settings for Oracle and Goldengate on Linux and UNIX sese 20 GoldenGate and Oracle Considerations esent ttntnntnnnntnns 21 Installing Goldengate for Microsoft SQL Server on Windows
11. TRANLOGOPTIONS Use TRANLOGOPTIONS to control datsbase specific aspects of log bssed capture For more information about using this parameter see the GoldenGste documentation cree ome Figure 10 22 Extract tranlogoptions parameter Director doesn t validate the parameters It appends the attributes to a single tranlogoptions line and doesn t dedupe for you You can insert one tranlogoptions item per line as shown here TRANLOGOPTIONS ASMUSER SYSQASM ASMPASSWORD AEBIZD ENCRYPTKEY DEFAULT TRANLOGOPTIONS ALTARCHIVELOGDEST ggs gg archive TRANLOGOPTIONS CONVERTUCS2CLOBS Generating Definition Files Generating definition files is similar to adding trandata options Without Director you need to use the DEFGEN utility to create a definition file To create a definition file in Director select the data source object in the diagram and choose Control View Details Then click the Definitions Files tab as Shown in Figure 10 23 www it ebooks info 207 CHAPTER 10 GOLDENGATE DIRECTOR Details for localhost_gg E Report Info Definitions Files Tran Redo Log Data Checkpointing DefGen allows you to produce the database definitions or metadata that allows transaction data to be moved between heterogeneous platforms Complete the fields below to produce and transfer the definitions files you will later use when configuring the system Source Transfer To User ID
12. Expert Oracle GoldenGate MASTER ORACLE S HIGH PERFORMANCE HETEROGENEOUS REPLICATION AND DATA VERIFICATION PLATFORM Ben Prusinski Steve Phillips and Richard Chung Apress For your convenience Apress has placed some of the front matter material after the index Please use the Bookmarks and Contents at a Glance links to access them tee 5 Apress www it ebooks info Contents at a Glance About the 1 xvii About the Technical Reviewer eere nnne nnne nnns xviii Acknowledgments 1 nere nnne nnn nnne nnn nnn xix Chapter 1 Introduction 1 Chapter 2 Installation ener nnn nnn nnns 5 Chapter 3 Architecture eere nnne nnn nnne nnne 33 Chapter 4 Basic Replication 49 Chapter 5 Advanced Features e nnenenn erreur uus 85 Chapter 6 Heterogeneous Replication 1ee eere 111 Chapter 7 Tuning ERU E
13. KK KK KK KKK KKK ad Running with the following parameters AK SK AAA AK KKK KA ok ok ok A ok ok Replicat RHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER TargetDB PASSWORD userpw HandleCollisions AssumeTargetDefs Map HR Target HR CACHEMGR virtual memory values may have been adjusted CACHEBUFFERSIZE 64K CACHESIZE 512M CACHEBUFFERSIZE soft max 4M CACHEPAGEOUTSIZE normal 4M PROCESS VM AVAIL FROM OS min 1G CACHESIZEMAX strict force to disk 881M Database Version Oracle Database 10g Enterprise Edition Release 10 2 0 4 0 64bi PL SQL Release 10 2 0 4 0 Production CORE 10 2 0 4 0 Production TNS for Linux Version 10 2 0 4 0 Production NLSRTL Version 10 2 0 4 0 Production Database Language and Character Set NLS LANG AMERICAN AMERICA AL32UTF8 NLS_LANGUAGE AMERICAN NLS_TERRITORY AMERICA NLS_CHARACTERSET AL32UTF8 For further information on character set settings please refer to user manual FRR ARK KR KK KK KK AK KKK KK KK KKK KK KK KKK KKK TUM Run Time Messages FRA AR AR A A A A A A A A A A A A AK A A A A OE A OK A OK A KE OK AK A A OK A A A OK A A A OK A A KK AK A OK AK A OK OK AK A OK OK OK KK Opened trail file dirdat 12000006 at 2011 01 28 23 19 11 87 www it ebooks info CHAPTER
14. eere 21 Installing Goldengate for Teradata on Windows and UNIX eere 21 Installing Goldengate for Sybase on Windows and UNIX eene 23 Installing GoldenGate for IBM DB2 UDB on Windows and UNIX nenne 23 Installing Oracle GoldenGate Director 119 eese 24 BEOE 25 Installing Oracle GoldenGate Director Server eee 27 Grant Database Privileges and Credentials to Oracle GoldenGate Director Server Schema 27 Install Oracle GoldenGate Director eese 27 Installing Oracle GoldenGate Veridata ennemis 27 GoldenGate Veridata Agent System Requirements sesenta 28 GoldenGate Veridata Agent Disk 28 GoldenGate Veridata Agent Memory Requirements sesenta 28 GoldenGate Veridata Agent Database Privileges essent 28 GoldenGate Veridata Server System Requirements sss 29 Install Oracle Goldengate Veridata eese 31 V ursus iM EE Kaia Ea 31 Chapter 3 Architecture ssssususnnennnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
15. Note GoldenGate uses a Commit Sequence Number CSN to identify and keep track of transactions processed by GoldenGate and to ensure data integrity The CSN is a GoldenGate platform independent representation of unique serial numbers that each DBMS uses to track the transactions it has processed For example in an Oracle database GoldenGate uses the Oracle System Change Number SCN to represent the CSN For a SQL Server database GoldenGate uses a concatenation of the virtual log file number a segment number within the virtual log and the entry number Extract writes the CSNs to the checkpoint and trail files that you can view using the Logdump utility You can use the CSN when starting the Replicat to begin processing at or after a specific CSN If needed you can examine the trail files in detail using the GoldenGate Logdump utility This is sometimes necessary for debugging purposes You can also use Logdump to filter records and save a subset of your trail file for special processing You learn more about the Logdump utility in Chapter 11 Troubleshooting Data Pump The data pump is another type of GoldenGate Extract process The data pump reads the records in the source trail written by the Local Extract pumps or passes them over the TCP IP network to the target and creates a target or remote trail Although the data pump can be configured for data filtering and transformation just like the Local Extract in many cases the dat
16. 212 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Alerts Oracle GoldenGate doesn t have any built in alerts If anything fails or hangs you won t know it if you aren t watching the system You have to write alert scripts manually as discussed in Chapter 8 or you can use Director With Director it s very easy to set up an alert To do so click the My Alerts tab on the main screen There are only two types of alerts Checkpoint Lag and Event Text in the ggserr 1og file Let s look at two examples First suppose you want an Event Text alert to check for any message containing ABEND Such messages should also appear in var log messages and the Windows Event log The AbendAlert alert entry should be similar to Figure 10 28 Parameters Oracle GoldenGate Log GGSCI My Alerts cae RESTE Alert Properties Handle this Alert using Built in Email Handler Send Test Email Custom Alert abendAlert Arg Name Value msgFrom Alert Type Jevent Text E IsmtpHost subjectLine Data Source iocathost gg lrecipList Process Name wildcard When Event is Errors Only and Event Text contains JABEND wildcard lt _ save delete Figure 10 28 Event Text alert Here are the steps to add this alert 1 In the Alert Type drop down list choose Event Text 2 Inthe Data Source drop down list
17. Checking Disk Space Even if disk space isn t an issue for you it s nice have a script running that monitors it You never know your disk space space may fill up in the next few years without your noticing You definitely need this script if trail file disk space is your weakest link Many scripts are available online for various systems so this section doesn t list one Google check disk space scripts to get the correct scripts for you operating systems You can write as many monitoring scripts as you want but you don t want to tax your systems All the scripts in this chapter are lightweight so they re safe to use 168 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE Summary This chapter has shown you how to set up a very good monitoring system for your Extract data dump and Replicat If you have many other GoldenGate processes that need to monitor then you just need to add them to your existing scripts Figure 8 1 includes 13 links in the GoldenGate processes but your configuration may have many more There are more ways that GoldenGate can break that you can imagine but the automated PL SQL lag monitoring script presented in this chapter can notify you about issues sometimes even before something major happens It s an independent process as long as the target database is up If the target database is down you don t get the hourly lag status e mail so you know something is wrong The alert can
18. Note The order of the parameters listed is important For example the REPLICAT parameter must be the first entry in the parameter file Next let s review the HANDLECOLLISIONS parameter HANDLECOLLISIONS You can use the HANDLECOLLISIONS parameter to have the Replicat attempt to resolve data errors or collisions when applying changes on the target A collision can happen when the target row is either missing or a duplicate The default processing for GoldenGate is not to automatically handle collisions which can be set using the NOHANDLECOLLISIONS parameter HANDLECOLLISIONS is used mostly during initial loads but it can also be used with care at other times for error resolution During the initial load you may run into errors with duplicate or missing data caused by the ongoing changes during the load For example if a row was updated and then deleted during the initial load GoldenGate may try to apply the update to the deleted row that was never loaded Another example is in problem situations where you need to have the Replicat reprocess transactions by repositioning it backward in the trail file If you add the HANDLECOLLISIONS parameter GoldenGate tries to resolve these errors automatically HANDLECOLLISIONS does the following when processing changes e Updates to rows that are missing are ignored e Deletes to rows that are missing are ignored e Inserts that are duplicated are changed to updates You should turn off HANDLECO
19. lt string gt Edit previous command HISTORY List previous commands OPEN FROM filename Open Log file RECORD REC Display audit record NEXT count Display next data record SKIP count FILTER Skip down count records FILTER Apply filter during skip COUNT Count the records in the file START time timestr END time timestr INT erval minutes LOG trail lt wildcard template gt FILE lt wildcard template gt DETAIL lt timestr gt format is yylyy mm dd hh mm ss www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR POSITION lt rba gt FIRST LAST EOF Set position in file REVerse FORward Set read direction RECLEN size Sets max output length EXIT QUIT Exit the program FILES FI DIR Display filenames ENV Show current settings VOLUME VOL V Change default volume DEBUG Enter the debugger GHDR ON OFF Toggle GHDR display DETAIL ON OFF DATA Toggle detailed data display RECLEN nnn Set data display length SCANFORHEADER SFH PREV Search for the start of a header SCANFORTYPE SFT Find the next record of TYPE typename typenumber lt filename template gt SCANFORRBA SFR Find the next record with lt SYSKEY gt lt syskey gt syskey 1 scans for next record lt filename template gt SCANFORTIME SFTS F
20. 195 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Adding an Initial Load Task Click the data source and choose Actions Add New Initial Load Task as shown in Figure 10 6 Figure 10 6 Choosing Initial Load Task You can fill in the Capture Name and Delivery Name in the Initial Load Task dialog Make sure the source and target database logins are correct Click OK Director creates all the necessary files for you as shown in Figure 10 7 Next you tell Oracle GoldenGate what tables you need to replicate with the initial load 196 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Oracle GoldenGate Director 11 1 1 0 0 ioj x File Edit Actions Controls Help 6S AOSR OBA B localhost gg Extract Task EXTINIT Map for EXTINIT _ 2j MEUM Replicst Task REPINIT Data Source tag localhost gg for REPINIT Lad Y 4 localhost 9g Add New Data Sources Parameters Oracle GoldenGate Log GGSCI My Alerts St localhost og Eo e manager manager 9000 c gasdirdatYa Figure 10 7 Files are created for the Initial Load task Mappings tell Oracle GoldenGate what you need to replicate over to the target database So you need to create mapping rules The guidelines for creating mapping rules are as follows e When configuring a mapping statement for a Capture object do
21. 2 After verifying there is no lag stop the data pump Extract on the standby database as shown in the following example GGSCI standbyserver LAG EXTRACT PHREMD3 Sending GETLAG request to EXTRACT PHREMD3 Last record lag 4 seconds At EOF no more records to process GGSCI standbyserver STOP EXTRACT PHREMD3 3 After verifying there are no more records to process stop the Replicat on the primary database as shown in the following example GGSCI primaryserver LAG REPLICAT RHREMD3 Sending GETLAG request to REPLICAT RHREMD3 Last record lag 6 seconds At EOF no more records to process GGSCI primaryserver STOP REPLICAT RHREMD3 Now that the replication from the standby to the primary is complete and stopped you can proceed with restarting the original replication from the primary database to the standby database in part 5 as shown in Figure 12 8 255 www it ebooks info 12 256 Database DISASTER RECOVERY REPLICATION Start the Extract Start the Pump Start the Replicat 1 3 4 PE 0070777 ra Local pg D Standby 1 Data Pump Replicat Database ee ee es Remote Trail Start the Application 2 Figure 12 8 Disaster recovery planned switchover part 5 The steps for Part 5 as shown in Figure 12 8 are as follows 1 Now you have completed Part 5 of your replication switchover process and the databases are back to their original
22. GoldenGate Veridata Agent Memory Requirements e 1GB or more of RAM memory is required for the Oracle GoldenGate Veridata Agent software GoldenGate Veridata Agent Database Privileges Depending on the specific database platform chosen for the Veridata comparison you will need to grant database level privileges to the Oracle GoldenGate Veridata Agent For example Oracle requires the following database privileges e GRANT SELECT on tables that will be compared by Veridata 28 www it ebooks info CHAPTER 2 88 INSTALLATION e GRANT CONNECT e SELECT CATALOG ROLE MS SQL Server requires the following database privileges for the Veridata Agent e VIEW DEFINITION for databases that are compared e db datareader on tables compared e SQL Server authentication must be used For IBM DB2 and Veridata agent you need to configure the SELECT privileges for all tables used in the comparison Teradata also requires SELECT privileges for tables used by Veridata as part of the comparison check GoldenGate Veridata Server System Requirements Disk Requirements Oracle GoldenGate Veridata has the following disk space requirements e of disk storage for basic setup Memory Requirements Oracle Veridata has the following memory requirements e 1GB or more of available system memory is required for the Oracle GoldenGate Veridata Server We recommend 64 bit OS for best performance with Oracle GoldenGate Veridata processing operations By
23. HELP HISTORY INFO ALL OBEY SET EDITOR SHELL SHOW VERSIONS note you must type the word COMMAND after the to display the help topic i e GGSCI sys1 gt help command For help on a specific command type HELP command object Example HELP ADD REPLICAT In the following example you need to find the syntax and description for the ADD EXTRACT command GGSCI oracledba 2 help add extract ADD EXTRACT Use ADD EXTRACT to create an Extract group Unless a SOURCEISTABLE task or an alias Extract is specified ADD EXTRACT creates checkpoints so that processing continuity is maintained from run to run Review the Oracle GoldenGate Windows and UNIX Administrator s Guide before creating an Extract group INFO The INFO command provides status information for the current Oracle GoldenGate configuration Some examples follow INFO ALL shows a complete overview of GoldenGate processing GGSCI oracledba 11 info all allprocesses Program Status Group Lag Time Since Chkpt MANAGER RUNNING EXTRACT STOPPED EATAA 00 00 00 1435 43 49 GGSCI oracledba 12 info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING EXTRACT STOPPED EATAA 00 00 00 1435 43 52 294 www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR INFO EXTRACT extract process group name displays a status report for Extracts as shown in the following example GGSCI oracledba 15 g
24. INDEX 326 SQLEXEC statements 108 srvctl utility 101 stalling Replicat process 236 standby data pump disaster recovery replication 249 250 standby Extract disaster recovery replication 248 249 standby Replicat disaster recovery replication 250 251 START command 228 START EXTRACT lt extract name gt command 219 statistics interim 286 STATOPTIONS parameter 89 STATS command 69 83 248 268 269 271 stats extract tablename command 141 146 stats replicat tablename command 141 146 STATS RHREMD1 command 130 STATUS command 295 STATUS EXTRACT command 226 Status of ABENDED 269 271 Status of RUNNING 269 271 Status of STOPPED 269 271 STOP EXTRACT command 68 Stop Job button Veridata Main screen 179 STOP state 164 storage disk tuning 149 150 fastest available 288 streams replication 3 4 supplemental logging database level enabling 54 55 verifying 54 table level enabling 55 57 SUPPLEMENTAL LOG DATA 54 SUPPLEMENTAL LOG DATA MIN 54 SUPPRESSTRIGGERS option 57 Sybase DBMS database management system to Oracle replication 123 on Windows and UNIX installation of Oracle GoldenGate 23 synchronization of data 238 239 syntax errors in parameter files 230 system requirements for Oracle GoldenGate database server versions 16 disk space 17 memory 16 for Microsoft Windows Clustered environments 18 network 17 operating system 18 www it eboo
25. Implementing Parallel Extracts and Replicats with Table Filtering In this example all of your interviews analysis and problem solving point to a slowdown in the Replicat so your first tuning change is to split the data pump Extract and Replicat into two parallel processing groups You do this to try to divide the replication workload make the Replicat run faster and reduce the Replicat lag For now in the example you split the GoldenGate process groups by filtering for specific SQL tables Later you see how to split the replication another way by using key ranges Note When splitting GoldenGate processing groups remember to keep tables related by SQL referential integrity constraints together in the same GoldenGate process groups Figure 7 3 shows more details on how you split the data pump Extract and Replicat for the HR schema 135 www it ebooks info CHAPTER 7 TUNING 136 Table Filter Employees Jobs Job History Locations Departments r Data Ed Replicat 1 qm D D 5 Trail 1 Dp 1 La 12 RHREMD1 Target Database Extract Tal 77777777 LHREMD1 qr Data 1 iil Replica rt D 7 D Trail 1i E Ga om Database Table Filter Countries Regions Figure 7 3 Parallel data pump Extract and Replicat groups for the HR example Local Extract LHREMDI remains unchanged and still extracts all the tables for the HR schema Usin
26. Note The CSN is GoldenGate specific each database system has its own way of tracking the commit sequence Oracle uses the System Change Number SCN DB2 2 05 uses the relative byte address RBA MySQL uses LogNUm and so forth 154 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE Why Monitoring the Extract Is Important In Oracle the Extract relies on the Oracle redo log to work Online redo logs are converted to archived logs after the last entry is written for a database in archivelog mode In some systems the DBA may compress and move the archived logs to other locations that aren t accessible to GoldenGate If the Extract is down for long period of time and the archived log isn t accessible then you lose data In mission critical environments you need to monitor your Extract process closely In normal database systems GoldenGate trail files use only around 30 or less of the overall redo log file The other 70 of the redo data are system related logs that aren t needed for GoldenGate So for example if you store the trail files for two weeks in case of a disaster the total trail files size for two weeks is daily total redo size x 0 3 x 14 You need around four times the total daily redo size to store the entire two weeks of change log data Two weeks of outage is very rare on major database systems Therefore as long as the Extract process can generate the trail files the Replicat will load them Getting
27. Performing the Initial Load 3 Starting the Extract Starting the Pump Nm Ps Replicat 0 Old gt Local Replicat gt 7 Database Extract Database Source Database Trail Application Figure 13 2 Zero downtime migration replication setup part 1 The example migrates the PAYROLL database to a new database software release In the Oracle GoldenGate parameter files you can use the PR abbreviation to indicate the payroll application In addition there are two tables that contain data that doesn t need to be migrated so you exclude those tables from the replication The example could apply in many different specific migration scenarios such as migrating from an Oracle 9i database on a Sun Solaris platform to an Oracle 11g database ona Red Hat Linux platform Keep in mind that whatever database or application you happen to be migrating the concepts and steps are similar to this example It can get more complicated if for www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION example you re migrating heterogeneous databases or doing complex application migrations In that case you may have more complex Oracle GoldenGate mapping or transformation filters and parameters but the overall process is still the same Oracle GoldenGate can handle those mappings and transformations as demonstrated in the previous chapters Because Chapter 4 covered steps 1 4 in detail let s move on to the detailed steps for part 2
28. Source Target Database Use the following process to implement one way at a time replication as shown in the figure 1 Start one way replication from the source database to the target database as shown in the bottom half of Figure 3 6 A Local Extract running on the source server extracts data and writes to the source trail 2 Adata pump is running on the source server This is optional but recommended The data pump reads data from the source trail and transmits it over the network to the remote trail 3 Replicat running on the target updates the target database Multiple parallel Replicats can be configured if needed to improve performance 4 Thereplication from the source to the target may continue for as long as needed For example with zero downtime database upgrades you continue replicating from the source to the target until you re ready to switchover to the newly upgraded database At this point the replication is reversed and the target database becomes the new source database as shown at the top of Figure 3 6 5 A Local Extract running on the source server extracts data and writes to the source trail 6 A data pump is running on the source server This is optional but recommended The data pump reads data from the source trail and transmits it over the network to the remote trail 7 AReplicat running on the target server updates the target database Multiple parallel Replicats can be configured if neede
29. www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION e Fast switchover from the primary database to the standby disaster recovery database for both planned and unplanned outages e After failover or switchover to the standby database the database roles are reversed and it becomes the primary database As needed the database roles can be reversed back to their original configuration at a later time e After failover or switchover changes to the standby database must be captured to allow for synchronization when the primary database is returned to the configuration In this chapter we will show how you can meet these requirements using GoldenGate For the examples we will use the HR schema as in the earlier chapters Disaster Recovery Replication Topology The underlying topology for disaster recovery is one way replication with some enhancements to support disaster recovery as shown in Figure 12 1 You may remember from Chapter 3 that one way replication is the simplest topology and is often used for reporting or query offloading purposes In our case the one way replication topology will be used for disaster recovery purposes Data is replicated from a single source database the primary to a single target database the standby in only one direction Changes to database data are only made at the primary database and then replicated to the standby database For disaster recovery a few enhancements are needed to the ba
30. CHAPTER ARCHITECTURE 36 You can instruct Extract to write any records that it s unable to process to a discard file for later problem resolution And you can generate reports automatically to show the Extract configuration You can set these up to be updated periodically at user defined intervals with the latest Extract processing statistics Source Trail The Extract process sequentially writes committed transactions as they occur to a staging file that GoldenGate calls a source trail Data is written in large blocks for high performance Data that is written to the trail is queued for distribution to the target server or another destination to be processed by another GoldenGate process such as a data pump Data in the trail files can also be encrypted by the Extract and then unencrypted by the data pump or delivery process You can size the trail files based on the expected data volume When the specified size is reached a new trail file is created To free up disk space you can configure GoldenGate to automatically purge trail files based on age or the total number of trail files By default data in the trail files is stored in a platform independent GoldenGate proprietary format In addition to the database data each trail file contains a file header and each record also contains its own header Each of the GoldenGate processes keeps track of its position in the trail files using checkpoints which are stored in separate files
31. Evolution and Oracle GoldenGate With the advent of true log based replication technology from Oracle and Quest SharePlex advances made it possible to finally perform real time based replication Out of the mainframe and midsize systems world arose a shining star in the development of real time transaction based replication techniques In the late 1980s a small software company called GoldenGate came up with a different method of replicating data between database platforms Until GoldenGate replication of data across platforms and vendors was problematic at best For example complex software development was required to harness the power of Oracle to non Oracle database environments via the Pro C and software development APIs to allow transactions to be moved between environments GoldenGate invented a powerful and novel way to replicate data while achieving high performance with accuracy Instead of using different formats GoldenGate implements a uniform format to perform data replication operations by using a command prompt interface called GoldenGate Software Command Interface GGSCI GGSCI commands are used to create new GoldenGate replication configurations as well as to perform administration tasks Committed transactions are stored in flat files called trail files on the filesystem for both the source and target systems In addition GoldenGate provides a seamless and transparent method to perform real time data replication with guaranteed tra
32. In part 2 of the initial setup you configure but don t start the replication back from the new database to the old database as shown in Figure 13 3 These steps are necessary to set up the replication for migration fallback if needed The three steps include setting up and configuring the Local Extract data pump Extract and Replicat These Extracts and Replicats are used later for the fallback scenario Adding the Adding the Adding the Fallback Replicat Fallback Pump Fallback Extract GRE m WE 36809 C 23 pg l Old pag Local Ka New Database Replicat lt a Data Pump i Extract Database 14 Remote Source Database Trail Trail Application Figure 13 3 Zero downtime migration replication setup part 2 Following are descriptions for steps 5 7 as shown in Figure 13 3 1 Configure and add the fallback Extract The fallback Extract is started later to capture changes after the migration cutover and used for fallback if needed 2 Configure and add the fallback data pump Extract which is started later after the migration cutover and used for fallback if needed 3 Add the fallback Replicat which is started later after the migration cutover and used for fallback if needed Let s look at each of the zero downtime migration replication steps in the next section beginning with setting up and starting the Local Extract Configuring the Local Extract for Zero Downtime Migration After you ve verified that the pre
33. O Optional download Oracle GoldenGate on Oracle v11 1 1 0 0 Media Pack for Microsoft Windows 32 bit Oracle Licenses and Options ip files included in the Media Pack xa EX GoldenGate V11 1 1 0 0 for Oracle 10g on Windows XP 2003 O select based on DB version and OS 008 pue GoldenGate V11 1 1 0 0 for Oracle 11g on Windows XP 2003 O select based on DB version and OS LEGEND See Legend below on details regarding table notation X Required downloads in order to run the associated products and licenses D Documentation Download the Client software if applicable O Optional download Figure 2 4 README instructions for Oracle Goldengate software installation Downloading from OTN For readers who want to set up a test demo environment to learn how to work with Oracle Goldengate software we recommend that you download the software from OTN as shown below at http otn oracle com Oracle allows you to use the GoldenGate software for educational purposes without a purchased license For use within a production environment you would download the software from the E Delivery site Figure 2 5 shows the Oracle Technology Network OTN website www it ebooks info 2 10 INSTALLATION Oracle Technology Network Woekd m b t yew Hutory Bookmarks Yaheei I 6 te com techeetwerhvindesx mri D H Mest Visted _ Getting Started i Latest o BA
34. fallbacks data pump configuring for zero downtime migration 272 273 Local Extract configuring for zero downtime migration 271 272 migration 276 278 Replicat process configuring for zero downtime migration 273 274 fetch failures 239 240 FETCHOPTIONS NOUSESNAPSHOT parameter 240 File Size field 222 file access problems 231 files definition generating 207 209 discard 223 225 missing 224 not opening 225 too big 224 parameter configuration issues with 229 230 trail changing size of 206 missing 228 opening with Logdump utility 299 300 problems with 221 223 filesystem directory 38 FILTER option Oracle GoldenGate Replicat parameter 222 INDEX FILTER parameter 96 filtering columns 95 96 rows 96 tables 94 95 135 141 Finished Jobs panel Veridata Main screen 180 183 first_change 157 FORMAT RELEASE option 62 FORMATASCII parameter 238 FORMATSQL parameter 238 FORMATXML parameter 238 FUNCTIONSTACKSIZE parameter 230 231 G generation rate of trail files 160 162 getlag command 159 160 GETREPLICATES parameter 108 gger ckpt table 108 gger ggs dirprm directory 59 77 gger ggs ggserr log directory 64 82 gger ggs oralO directory 282 GGSCI GoldenGate Software Command Interface 46 189 190 GGSCI command 16 121 284 285 293 GGS GET LAG 163 311 www it ebooks info INDEX GHDR ON command Logdump 299 GLOBALS file 91 103
35. port 7840 AutoStart ER AutoRestart ER PurgeOldExtracts gger ggs dirdat UseCheckpoints MinKeepDays 2 Making the Replication More Secure This section discusses a couple of items to make your GoldenGate replication environment more secure First you look at encrypting the GoldenGate passwords and then you examine encrypting the GoldenGate trail files Of course turning on these features is optional and depends on your specific security requirements Encrypting Passwords You should encrypt the GoldenGate database user passwords in your parameter file in any secured environment The following example shows how to encrypt the password abc and specify it in the parameter file The example uses a default encryption key but you can use a specific key if needed GGSCI sourceserver 1 encrypt password abc No key specified using default key Encrypted password AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB When you have the value for the encrypted password you can update your parameter file with the value USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default In addition to encrypting passwords you can encrypt the trail files as you see next www it ebooks info CHAPTER 5 ADVANCED FEATURES Encrypting the Trail Files In some cases you may want to encrypt the data stored in the trail files You can do this by adding the ENCRYPTTRAIL and DECRYPTTRAIL commands to your parameter files Let s look at
36. www it ebooks info CHAPTER 6 88 HETEROGENEOUS REPLICATION Checkpoint files Process status files SOL script files Database definitions files Extract data files Temporary files Veridata files Veridata Lock files Veridata Out Of Sync files Veridata Out Of Sync XML files Veridata Parameter files Veridata Report files Veridata Status files Veridata Trace files Stdout files ggs mssqlserver dirchk created ggs mssqlserver dirpcs created ggs mssqlserver dirsql created ggs mssqlserver dirdef created ggs mssqlserver dirdat created ggs mssqlserver dirtmp created ggs mssqlserver dirver created ggs mssqlserver dirver lock created ggs mssqlserver dirver oos created ggs mssqlserver dirver oosxml created ggs mssqlserver dirver params created ggs mssqlserver dirver report created ggs mssqlserver dirver status created ggs mssqlserver dirver trace created ggs mssqlserver dirout created NANIANANANANNANAANAN GGSCI oracledba 2 gt Windows Vista and Windows 2008 security options need to be configured to allow the manager and Extract processes to be started Otherwise you will receive the following error message when you try to start manager GGSCI oracledba 7 gt start mgr Process creation error WIN32 API CALL CreateProcess failed 740 The requested operation requires elevation The solution is simple run the command msconfig utility from a Windows shell prompt as shown in Figure 6
37. By adding it now before it s needed you re ready to start it when needed First create a parameter file for the Replicat You re configuring the Replicat to apply all the DML changes from the newly migrated PAYROLL database back to the old database in case of fallback Notice that you removed the HANDLECOLLISIONS parameter from the fallback Replicat You shouldn t have any collisions at this point because the databases are already synchronized Let s look at the Replicat parameters for RPREMP2 as shown in the following example GGSCI newserver 1 edit params RPREMP2 Replicat RPREMP2 273 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION 274 USERID GGER PASSWORD userpw ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt RPREMP2 dsc Append DiscardRollover at 02 00 AssumeTargetDefs Map PAYROLL Target PAYROLL The next step is to add the Replicat group Here s you re adding the RPREMP2 Replicat to the Oracle GoldenGate configuration GGSCI oldserver ADD REPLICAT RPREMP2 EXTTRAIL dirdat p4 Remember you won t start the fallback Replicat until after the application migration cutover to the new database You re now finished with the initial setup for the zero downtime migration replication All the steps you ve completed so far are in preparation for the application cutover With old and the new databases synchronized you re prepared for the cutover to the new datab
38. Checkpoint Lag 00 00 00 updated 00 00 07 ago Log Read Checkpoint File c ggs_ src trails aa000000 First Record RBA 0 Extract Source Begin End c ggs_src trails aa000000 Initialized First Record c ggs_src trails aa000000 Initialized First Record Current directory C ggs trgt Report file C Nggs trgt dirrpt REPORA rpt Parameter file C ggs_trgt dirprm REPORA prm Checkpoint file C ggs_trgt dirchk REPORA cpr Checkpoint table GGS CHECKPOINT Process file C ggs_trgt dirpcs REPORA pcr Error log C ggs_trgt ggserr log 237 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 238 Note The convchk utility causes the checkpoint file to become the new master for Replicat checkpoints The Replicat resynchronizes the truncated checkpoint table with data from the checkpoint file Replicat Fails on Large Transactions If the target system can t support large transactions the Replicat process groups will fail with an out of memory condition To solve this error you can set the MAXTRANSOPS parameter to split up the Replicat transactions into smaller transactions or use multiple Replicat groups to process these in parallel Incompatible Record Errors with a Replicat One common problem during Replicat processing on the target system occurs when incompatible records bring the Replicat to its knees and failure In the GGSCI logs you see an error similar to the following 2011 03 09 11 00 22 GGS ERROR 509
39. Describe the existing or planned source and target databases What are the software releases for the database and operating system Are the servers stand alone or clustered How is the data storage configured What are the character sets for the source and target databases e Data currency How quickly and how often does the data need to be updated Can any lag in the data replication be tolerated If no lag can be tolerated and there is a high volume of changes you may need to devote more time in the project to designing and tuning the replication to avoid any lag Keep in mind that often reporting systems can tolerate lag and the target data doesn t need to be completely up to date with the source e Data volume How much data needs to be replicated How often is it updated You may be able to examine the existing database transaction log files to determine the amount of data change that is occurring in the source database The data change volume impacts the network bandwidth requirements between the source and target and the amount of disk space required for trail files e Data requirements You should understand any unique or special data requirements so they can be handled properly in GoldenGate For example if there are triggers they may need to be disabled in the target database Sequences need special parameters added if they need to be replicated Certain data types may need special handling or may not be supported For example in Gol
40. Filtering Tables You may remember from Chapter 4 that you instructed GoldenGate to extract all the changes for all tables in the HR schema using the following Extract parameter Table HR You did this by using the wildcard character asterisk after the schema name HR which tells GoldenGate to extract all the tables in the HR schema www it ebooks info CHAPTER 5 ADVANCED FEATURES Now suppose your business requirements have changed and you don t need to extract all the tables in the HR schema You only need to extract the EMPLOYEES and JOBS tables You can do this by changing the Extract parameters as shown in the following example Table HR EMPLOYEES Table HR JOBS You removed the wildcard and added only the two tables you want to replicate You may already be wondering whether you can accomplish the same thing by filtering using the Replicat instead of the Extract The answer is yes although there may be advantages to filtering using either the Extract or the Replicat or both it depends on your requirements and specific needs For example you may have multiple Replicats as targets that require different sets of data from different tables In that case it may be better to extract the data for all the tables and use each Replicat to filter out only the data it needs Another option is to configure a data pump Extract to do the filtering and send each Replicat only the data it needs On the other hand if you only need the
41. It should be specified in the Local Extract and Replicat parameters Next let s discuss the PASSTHRU parameter PASSTHRU You can specify the PASSTHRU parameter on the data pump if you aren t doing any filtering or column mapping and your source and target data structures are identical For GoldenGate to consider the tables identical they must have the same column names data types sizes and semantics and appear in the same order Using PASSTHRU improves performance by allowing GoldenGate to bypass looking up any table definitions from the database or the data definitions file You may also find this helpful if you need www it ebooks info CHAPTER 4 M BASIC REPLICATION to put your data pump Extract on an intermediate or middle tier server that doesn t have a database running RMTHOST Use RMTHOST and MGRPORT to tell the data pump Extract the remote server name and the port on which the GoldenGate manager is running This is the destination location where the data pump sends the local trail file over the TCP IP network You should verify that you have network connectivity between your source server and the MGRPORT on the RMTHOST If needed you can specify multiple remote hosts in the same parameter file to send trails to different servers RMTTRAIL You can use RMTTRAIL to specify the two character remote trail name for your data pump In this case you use 12 as the remote trail name Remember that 11 is the local trail The dat
42. Jog_director2 Password eA g tP ETE File Name C g9s dirdef filename def Overwrite Existing File Ta Refresh Table List Transfer Definitions Figure 10 23 Sourcedef and Targetdef file generation Enter the Database Access information for each table you need to add to the source target definition files Select one or more tables in the Select Table box and click Transfer Definitions when you re finished The tables automatically transfer to the target source location via the Manager process You may have more than one definition files for each schema on either the source or target if so concatenate the files or append them to the existing sourcedef file as shown in Figure 10 24 208 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Insert Parameters report n jreportcount ireportrollover rmthostoptions jrmtname rollover serializesource E sourceisfile sourceistable specialrun sglexec statoptions targetdefs anam zi SOURCEDEFS Use SOURCEDEFS to specify the name of a file on the target system that contains definitions of the source tables Source definitions sre required when using GoldenGste in s heterogeneous synchronizstion environment To generste the source definitions file use the DEFGEN utility which can be sctivated from Director see the online help Transfer the file to the target system before starting
43. QUERY select count empduplicate from HR EMPLOYEES where employee id and amp employee timestamp amp PARAMS 1 employee id p2 employee timestamp BEFOREFILTER ERROR REPORT amp TRACE ALL amp FILTER checkemployee empduplicate 0 ON UPDATE RAISEERROR 90000 This example executes a GoldenGate filter before replicating an update to the row The filter determines whether there are any rows already in the database with the same key and a newer timestamp than the row you re replicating If no rows that are duplicates with a newer timestamp are found checkemployee empduplicate 0 then you replicate the row If there are already rows with a newer timestamp checkemployee empduplicate 0 then you raise an error keep the existing row in the database and discard the row being replicated Refer to the Oracle GoldenGate Windows and UNIX Administrator s Guide for additional examples of conflict resolution routines Summary This chapter covered how to add advanced features to your basic GoldenGate replication configuration You saw how to enhance your basic reporting capabilities and make replication more secure using encryption You added some automatic start parameters to the Manager so your replication stays up and running longer You also reviewed how to include advanced table row and column mapping and data filtering in your replication configuration to meet your complex business requirements Finally the
44. Replicat Fails on Large TranSactions sssssssssssessesssssesssssssessssssssessssesssessssesssssssesseesssnsseseeaseneens 238 Incompatible Record Errors with a Replicat csssssssssssssssssessssessessessenseeseesessersensensenseseeneoees 238 www it ebooks info xiii CONTENTS Data Synchronization 1 5 238 Tables Missing Key 239 Character Set Configuration Issues sesenta 239 Missing Column 239 Fetch Failures 239 Ir A EERTE 240 Chapter 12 Disaster Recovery Replication 241 Mil 241 Ll e MM 242 Disaster Recovery Replication 243 o 244 Configuring the Local Extract for Disaster Recovery eeeeeeeeeee ener nennen
45. STOP EXTRACT IHREMD1 Verifying the Initial Load You can verify the initial load by viewing the GoldenGate report file on the target server Following is an excerpt from the report file that shows the number of rows that have been loaded GGSCI targetserver gt VIEW REPORT DHREMD1 A EK kkk kk Run Time Statistics 2K 2K OK OK 2 SK HE 2 OK OK OK SE OK OK SE E OK OK OK OK HERE SE SE SE SE SK SE SE OK SK OK OK www it ebooks info CHAPTER 4 M BASIC REPLICATION Report at 2011 01 21 23 31 42 activity since 2011 01 21 23 31 36 From Table HR COUNTRIES to HR COUNTRIES inserts 25 updates 0 deletes 0 discards 0 From Table HR DEPARTMENTS to HR DEPARTMENTS inserts 27 updates 0 deletes 0 discards 0 From Table HR EMPLOYEES to HR EMPLOYEES inserts 117 updates 0 deletes 0 discards 0 From Table HR JOBS to HR JOBS inserts 19 updates 0 deletes 0 discards 0 From Table HR JOB HISTORY to HR JOB HISTORY inserts 10 updates 0 deletes 0 discards 0 From Table HR LOCATIONS to HR LOCATIONS inserts 23 updates 0 deletes 0 discards 0 From Table HR REGIONS to HR REGIONS inserts 4 updates 0 deletes 0 discards 0 You need
46. choose localhost gg or your data source that has the Manager process running 3 Inthe Process Name text box enter for all processes Or to use one process type its name such as EXTGGSI 4 IntheWhen Event Type Is drop down list choose Errors Only You re creating an ABEND alert and abend is an error 213 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR 5 Inthe And Event Text Contains text box enter ABEND This is the actual text that appears in the ggserr log and var log messages 6 Inthe msgFrom Value field enter anything you want so you know who sent the alert 7 Inthe smtphost field enter any SMTP mail relay server 8 Inthe subjectline field enter something like EXTGGS1 has abended 9 Inthe reciplist field enter the e mail addresses of the alert recipients Next let s set up a Lag alert To do so enter the information shown in Figure 10 29 Figure 10 29 Lag alert for Extract EXTGGS1 Director doesn t need a server login because it handles all the requests via the Manager process on the server side So if the Manager is down you don t get any alerts Refer to the chapter 8 for more advanced topics 214 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Summary By now you should be familiar with the Director processes Director is a cross platform tool so you can use it for other databases and platforms with some minor changes The differences are handled by Dire
47. gt START EXTRACT LPREMP2 2 Prepare the new database for the application and start the application connected to the new database In preparation you may need to grant some SQL permissions to your application users on the new database Also remember to enable any triggers or cascade delete constraints you may have stopped while the new database was the replication target 3 Start the data pump Extract the new database as shown in the following example GGSCI oldserver gt START EXTRACT PPREMP2 4 Start the Replicat on the old database as shown next Make sure to disable any triggers or cascade delete constraints on the old database before processing the Oracle GoldenGate changes from the new database GGSCI newserver gt START REPLICAT RPREMP2 Performing the Migration Fallback After completing the migration you may decide you need to revert back to your original database due to problems or for other reasons This section covers the steps necessary to fall back These steps are divided into two parts in part 1 you stop the replication from the new database back to the old database see Figure 13 6 and in part 2 you start the replication back up again from the old database to the new database in preparation for the next cutover attempt Stop the Replicat Stop the Pump Stop the Extract m e EN 5 pb TAS us Old pa Local New Database Replicat a Data Pump Extract Database En
48. www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE side to the target side You also need to do the same thing on the target side for monitor points 7 to 13 If you add an additional data pump process then you need to add a monitor to points 2 3 4 and 7 on the source side and the corresponding points at the target However this is the logical view The actual number of scripts may vary For example one script may have a birds eye view of GoldenGate s overall health Potential Golden Gate Monitor Points on One Way Replication Manager Extract Data Pump Processes lt 7 3 Manager Replicat Collector Processes FF 3 8 9 Server Server Storage Devices Memory Devices 5 12 1 Source Target Database Database Figure 8 1 Potential GoldenGate monitor points Each Extract process has a database connection querying the database dictionary to check if there is any change If changes occur extract process reads the online redo log to capture the changes and writes to the trail file If the data isn t in the online redo log it reads the archived log If the data isn t in the archived log extract process reads the alternative archived log if you specify ALTARCHIVELOGDEST lt path name gt in the Extract parameter files If extract process still can t find the transaction with the correct Commit Sequence Number CSN extract process abends and tells you the missing file sequence number
49. you will need to log in to the source Oracle database and turn on supplemental logging for the EMP and DEPT tables for the SCOTT user schema Supplemental logging must be enabled within the database environment so that changes made to data in the tables are captured during DML operations such as whenever Insert or Updates are performed against the tables for Oracle ggsci GGSCI DBLOGIN USERID GGS GGSCI ADD TRANDATA GGS EMP GGSCI ADD TRANDATA GGS DEPT Next you will need to verify that supplemental logging has been turned on for these tables 113 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION GGSCI gt INFO TRANDATA GGS Logging of supplemental redo data enabled for table GGS DEPT 2011 04 03 22 31 21 WARNING OGG 00869 No unique key is defined for table EMP 11 viable columns will be used to represent the key but may not guarantee uniqueness KEYCOLS may be used to define the key Logging of supplemental redo data enabled for table GGS EMP This warning is very important to keep in mind when you set up the GoldenGate environment since GoldenGate requires either a primary or unique key to be defined for tables in order to identify unique key columns as part of replication processing Otherwise all of the columns will be used thereby affecting performance adversely Configure Source Definition on Source Oracle Database System The next step is to perform the source definition file creation mapping with the fol
50. 00 00 00 Lag 00 05 localhost 9g 5 Trail C ga4C ggs dirdat c2 Map for REPGGS Replicst REPGGS localhost gg localhost gg Lag 00 00 00 Figure 10 21 Complete flow with the Data Pump process Start the Extract data pump and Replicat You may need to restart the Oracle GoldenGate d client in order to see the diagram with the proper green lines The Refresh Diagram feature doesn t work sometimes Now that everything is running it s a good time to check out the View Detail features for the Extract Replicat and data source icons Right click each icon and select View Detail Doing so executes some GGSCI commands for you such as View report group name gt info group name gt lag extract group name send group name and so on Additional Director Features and Tricks There are many other Director features that aren t covered in the previous sections This section goes over some key features and tricks After reading this section you ll know what Director can do for you compared to the Oracle GoldenGate GGSCI command line Alter Extract or Replicat RUN Options Director allows you to alter the Extract Replicat RUN options For example suppose you want to reset the Extract begin timestamp or Replicat seqno RBA To do so click the Extract Replicat object in the diagram and choose Controls Run Options The Extract Replicat must be stopped in order for you to adjust these options It s important to understand the proc
51. 127 Chapter 8 Monitoring Oracle GoldenGate 153 Chapter 9 Oracle GoldenGate Veridata rne 171 Chapter 10 GoldenGate Director nennen nnne 189 Chapter 11 Troubleshooting Oracle GoldenGate 217 Chapter 12 Disaster Recovery Replication eee 241 Chapter 13 Zero Downtime Migration Replication 263 Chapter 14 Tips and Tricks 279 Appendix Additional Technical Resources for the Oracle GoldenGate Administrator ees 291 Lj quem 91 www it ebooks info 1 Introduction Oracle Corporation acquired GoldenGate in 2009 as part of future strategy to implement advanced replication technologies within the product suite of data warehousing and real time transaction management Before the advent of Oracle GoldenGate technology data replication was performed using Oracle Streams and third party replication solutions such as Quest SharePlex This chapter discusses various types of database replication methods along with the techniques used by each method to illustrate how Oracle GoldenGate came to be the logical conclusion for the current and future real time based data transaction replication tech
52. 159 160 recording lag status in ggserr log file 162 trail file generation rate 160 162 Write Checkpoint command 160 recording status in ggserr log file 162 LD_LIBRARY_PATH variable 230 LHREMDI Extract group 60 LHREMP2 local extract 246 LIMITROWS option 239 Linux GoldenGate Process failures 222 installation of Oracle GoldenGate 20 listener ora file 102 listener ora network configuration 220 listeners updating 102 literal comparison 180 loading 70 75 with DBMS utilities 76 initial adding Replicat group 74 configuring Replicat parameter file 72 74 INDEX Extract parameter file 71 prerequisites for 70 71 starting 74 verifying 74 75 Local Extract configuring 59 62 Extract parameter 60 EXTTRAIL parameter 62 SETENV parameter 61 TABLE parameter 62 USERID parameter 61 62 configuring for zero downtime migration 267 268 disaster recovery replication 245 246 fallback configuring for zero downtime migration 271 272 process 35 36 stopping 274 276 Local Extract LHREMD1 142 Local Extract trail file name 268 Local Extract trail 11 139 Local Extract trail 12 144 145 log based 2 Logdump utility commands and syntax for troubleshooting 296 300 accessing Logdump utility 296 getting help with Logdump syntax 296 299 315 www it ebooks info INDEX HISTORY command 299 opening trail files with Logdump utility 299 300 logging supplemental database leve
53. 186 SUMMAY 188 Chapter 10 GoldenGate Director nennen nnne nnn 189 Director COmpon erls isinincniinicerinininin irs niin incertus xk FEES n UE EHE RE Fina Enn ni 189 GoldenGate GGSCI Instances eene tnnt tnnt tnnt tentent 190 Director Administrator eese terrent 190 Director Server and Database sse enenatis 190 www it ebooks info CONTENTS DIRCCUOR WD 191 Director 192 Setting Up the Data Source 192 Modifying the Manager Parameter file eere 193 Modifying the Parameter File Using the Built in Editor eere rene 193 Modifying the Parameter File Using the GUI eeeeeeeeeeeree neret 194 Setting Up the Ir ni LEE To eme 195 Adding an Initial Load Task eene tnes 196 Creating a One Way Replication esee 200 Adding a Data Pump Process 204 Additional Director Features and Tricks 206 Alter Extract or Replicat RUN Options cssssssssssssssssssssssessessecsesensensensessensensensensensansansensanease 206 Changing the Trail File Size
54. 226 Incorrect Software Versions with Oracle GoldenGate essen 227 Database Availability ISSUCS eese tnnt tnnt tnn 227 Missing Oracle GoldenGate Process GrOUpS ssssssssssesssssessessessssssesssesessessessssesseesessesnssnssesseesseneans 228 Missing Oracle GoldenGate Trail Files eene 228 Oracle GoldenGate Parameter File Configuration Issues eene 229 Operating System Configuration Issues with Oracle GoldenGate ss 230 Network Configuration Issues with Oracle GoldenGate eene 231 Network Data Transfer ISsues esee nennen tnnt nnne tintas tete 232 Oracle Database Issues with GoldenGate eene 234 Extract Can t Access Oracle Database Archive and Redo Logs esses 235 Extract Failure Conditions Due to Oracle Source Database lIssues eee 235 Data Pump EITOIS cnet 236 Replicat Errors on the Oracle Database Target System senes 236 Replicat Hangs on the Target System eese rennen nennen 236 Replicat Experiences an Abend Failure on the Target 236
55. 4 System Configuration NNNM A General Boot Services Startup Tools Startup selection Normal startup Load all device drivers and services D Diagnostic startup Load basic devices and services only Selective startup V Load system services lll Load startup items mmm mI rm Figure 6 4 Security settings for manager on Windows and Microsoft SQL Server 117 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION Now you should be able to start the manager process on Windows for the source system from the services menu as the Administrator account C ggs mssqlserver gt ggsci Oracle GoldenGate Command Interpreter for ODBC Version 11 1 1 0 0 Build 078 Windows optimized Microsoft SQL Server on Jul 28 2010 18 55 52 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved GGSCI oracledba 1 gt info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING GGSCI oracledba 2 gt status mgr Manager is running IP port oracledba 2000 GGSCI oracledba 3 gt view params mgr PORT 2000 Creating Sample Microsoft SQL Server Database Now that we have the Oracle GoldenGate software installed on both source and target systems we need to create and populate the target Microsoft SQL Server database From Microsoft Windows under the Start gt Programs run Microsoft SQL Server Enterprise Manager You will then need to expand out the console
56. 8 TROUBLESHOOTING ORACLE GOLDENGATE Replicat group you need to log into the target database with the DBLOGIN command before removing the checkpoint from the table and Replicat deletion You can use the following syntax to perform this task DBLOGIN USERID user PASSWORD pw DELETE REPLICAT group Trail Files that Purge Too Soon Another painful situation with trail files occurs when the trail files are purged too soon This causes errors due to missing data between the source and target systems One root cause to check for to solve this problem is multiple Replicats and data pumps reading from the same trail file Also check to see if you re using the PURGEOLDEXTRACTS parameter details The key thing is to realize that you should only use this parameter as a Manager parameter you shouldn t use PURGEOLDEXTRACTS as an Extract or Replicat parameter When you use PURGEOLDEXTRACTS as a Manager parameter you can use the option USECHECKPOINTS to defer purging of the trail file until all of the Oracle GoldenGate processes have finished processing data for the trail file In addition you can use the MIN option to store the trail file for a set period of time Whenever you use the MIN option you must also set the MAX option for PURGEOLDEXTRACTS to denote the maximum amount of time to keep the trail file Oracle GoldenGate Error Log Analysis Fortunately Oracle GoldenGate is a well instrumented software product that provides a superb error l
57. About the Technical Reviewer Arup Nanda arup proligence com has been an Oracle DBA for over 16 years working on all aspects of Oracle database management from modeling to performance tuning He has co authored four books published 300 articles and presented over 150 sessions at various technology conferences he also received DBA of the Year in 2003 from Oracle He is an Oracle ACE Director a frequent blogger arup blogspot com and member of the Oak Table Network and the SELECT Journal Editorial Board He lives in Danbury CT xviii www it ebooks info Acknowledgments I would like to thank my co authors Steve and Richard for collaborating with me on this book Without their help this book would not have been possible I also would like to thank our technical reviewer Arup Nanda for his insight and time reviewing our book to ensure technical accuracy and quality In addition my thanks go out to the team at Apress including Jonathan and Anita for their support and insight Last but not least I want to dedicate this book to all fellow Oracle professionals who work hard in the trenches to support mission critical systems running Oracle and GoldenGate My appreciation extends to my parents for motivating me to become a writer in addition to many pursuits recommended Ben Prusinski I would like to thank my wife Kathy and sons Ryan and Jacob for their support encouragement and understanding while I took time away to wr
58. After Veridata is installed you can use the following guide to use your own data or the data we use in this chapter The high level steps are as follows 1 gm 00 Create database connections source and target Create group a group of comparing pairs Create compare pairs what tables to compare Create job to run one or more groups Create profile how you want it to run This is optional as you can use the default profile Run job Creating Database Connections After you log in to the GoldenGate Veridata Web click the Connection Configuration on the left panel as shown in Figure 9 2 then fill in the following information for both source and target Make sure GoldenGate Veridata agents are installed on both database instances You can either use the Wizard to set up the connection or use the setup screen as shown in Figure 9 2 Use defaults for other options for now 175 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA ORACLE GoldenGate Veridata M 2 SZ Logout About Help Home Connection Configuration gt gt Edit Connection ord source Running Jobs View By Job View By Group Save View By Compare Pair Datasource connection was successful Finished Jobs Connection Setting Connection Proper View By Job View By Group Name lord source View By Compare Pair Configuration Description Connection Configuration Group Configuration GoldenGate Connection Job Configuration Host
59. Collector process The delivery function reads the remote trail and updates the target database Each of the components is managed by the Manager process You may notice that the data pump and the Collector process are formatted slightly differently than the others The data pump is surrounded by dots to indicate that it s an optional process The Collector is bordered with dashes to show that it s a dynamic process that starts automatically and runs in the background I ll explain why and go into more detail about those and the other components in the next sections GoldenGate Components Figure 3 1 illustrates the various components you re most likely to work with Each of the components serves a different purpose in the overall GoldenGate configuration In this section you learn about the purpose of each component and how it fits into the overall replication flow In later chapters you learn how to set up and configure each of the components Let s now look at the components in more detail Source Database You may recognize that the source database isn t really a GoldenGate component but rather a vendor database that is part of your existing infrastructure GoldenGate supports a wide variety of heterogeneous source databases and platforms Table 3 1 shows the databases GoldenGate supports as source databases as of GoldenGate version 11g Note Be sure to check the GoldenGate product documentation for the most current list of support
60. DDL TIME yyyy MM DD hh24 mi ss RTRIM TIMESTAMP RTRIM STATUS RTRIM TEMPORARY RTRIM GENERATED RTRIM SECONDARY NAMESPACE RTRIM EDITION NAME FROM 2 TABLE TARGET X Comparing Incremental Data for Large Tables If you have a table with billions of rows you cannot afford to do a full table scan every time you need to compare Instead you can use row partition features to compare Veridata row partition not the database partition This requires nothing more than adding a sql predicate to the WHERE clause which if added correctly will make a big difference in terms of performance For example you can add SEQNO 1 to both the source and target row partition in the Compare Pair Configuration Screen and the Veridata will generate the SQL as follows SELECT SEQNO OWNER OBJECT NAME SUBOBJECT NAME OBJECT ID DATA OBJECT ID OBJECT TYPE TO CHAR CREATED yyyy MM DD hh24 mi ss TO CHAR LAST DDL TIME yyyy MM DD hh24 mi ss TIMESTAMP STATUS TEMPORARY GENERATED SECONDARY NAMESPACE EDITION NAME FROM VERIDATA2 BIG TABLE TARGET X WHERE SEQNO 1 181 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA 182 If you need to compare the data replicated in one day for example change_date is the column that you use to track the change then you can put this to the sql predicate statement in both source a
61. Delivery 87 Figure 10 24 Inserting a source or target definition into the parameter file Be sure to add the sourcedef targetdef file to the Extract or Replicat parameter file Use the Insert Parameter option on the Parameters tab in the Extract or Replicat to do this Finding Parameters or Attributes in Director Director doesn t implement all the parameters in its GUI interface including TRANSMEMORY and many others You have to use Edit Param feature to add these parameters manually In some cases attributes are also missing such as FETCHOPTIONS NOUSESNAPSHOT To remedy this situation you can add FETCHOPTIONS and manually enter NOUSESNAPSHOT in the text box Oracle GoldenGate also compresses parameters The nocompressupdates parameter doesn t appear in the list but if you choose compressupdates you can then pick nocompressupdates in the drop down list as shown in Figure 10 25 209 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR 210 Insert Parameters x allowlargetable a compressupdates nocompressupdates pe compressupdates ars compressupdates umi oe cachemar icheckparams icheckpointsecs cmdtrace o jcuserexit Idboptions ddi COMPRESSUPDATE S HOCOMPRE S SUPDATE S Use COMPRESSUPDATES and NOCOMPRESSUPDATES to control whether or not GoldenGste compresses updates for processing COMPRESSUPDATES causes only the key
62. Environments Before we can set up the Oracle GoldenGate process groups for Extract data pump and Replicat on the source and target environments we need to load the sample data on both source and target systems This is called the initial data load and unfortunately Oracle GoldenGate has no built in procedure to automate this task You can use a variety of vendor supplied utilities for Oracle and Microsoft SQL Server For Oracle you can perform an export import or data pump export and import For Microsoft SQL Server you can use the Microsoft native BCP and DTS bulk load utilities or perform a direct load with Replicat Source Oracle Database Configuration Before you can replicate data to the target Micrososft SQL Server database environment there are a number of preliminary setups that must be performed Let s walk through each of these procedures Create the Manager parameter file on the source Oracle system and specify the port that it should use cd ggs source ggsci GGSCI EDIT PARAMS MGR GoldenGate Manager parameter file PORT 50001 Start the Manager process GGSCI gt START MGR Verify that the Manager process has started GGSCI gt INFO MGR Manager is running IP port oracledba 50001 Log in to the Oracle source database as the scott user Create and populate the SCOTT EMP and SCOTT DEPT tables with sample data Perform a CREATE TABLE AS SELECT to copy the table data to the GGS user schema Next by using GGSCI
63. GoldenGate February 2010 Oracle GoldenGate high availability using Oracle Clusterware March 2010 The next section provides a quick guide at frequently used Oracle GoldenGate commands Quick Guide to Oracle GoldenGate Commands For the beginner who s new to Oracle GoldenGate as well as the experienced practitioner Oracle GoldenGate commands present a complex jungle to master This section shows you the commonly used commands to aid you in your quest to perform tasks in Oracle GoldenGate It will also serve as a reference if you need to look up the syntax for a command ADD Syntax ADD EXTRACT REPLICAT www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR The ADD command performed in the GoldenGate Software Command Interface GGSCT is the method for adding new configurations to Oracle GoldenGate You can use it to add new Extract Replicat or data pump groups as well as to add local or remote trail files on the source or target system for an Oracle GoldenGate configuration Examples ADD EXTRACT extora RMTHOST prod MGRPORT 7800 RMTNAME prodfin ADD REPLICAT repora EXTTRAIL d ggs dirdat rt GGSCI GGSCI is the command to enter the command line interface used to perform most administrative tasks for Oracle GoldenGate In the simplest form you open a shell prompt and type GGSCI from the Oracle GoldenGate base installation directory You can also set the path variable
64. GoldenGate Delivery for Oracle REPORA prm Extract read Incompatible record in C GGS_TRGT DIRDAT AA000000 rba 5466 getting header This occurs if the data gets corrupted in transit from the source system to the target system Another cause is a data format that isn t compatible with Replicat processing If you use the FORMATASCIT FORMATSQL or FORMATXML parameter for the Extract during the initial load it will cause the failure to occur Incompatible record errors with the Replicat also occur due to misconfiguration of trail files that are sent from the source to the target system when the Extract is configured to overwrite an existing trail file For instance if you have two Extract processes writing to the same trail file they will overwrite the one the Replicat is looking to use thus causing an incompatible record error The Replicat needs complete records to apply the data successfully from the trail file to the target system Sometimes the Extract and Replicat groups are dropped and rebuilt with the exact same trail file name This causes the Extract process to begin writing again at the first part of the same trail file causing issues with stepping on the trail file You can use the Logdump utility to investigate error 509 conditions with Oracle and GoldenGate In the event that you can t salvage the data the best solution is to resynchronize the target database with a new initial load and to add new trail files correctly to avoid futur
65. In the example you use wildcarding to map all the tables in the HR schema on the source database to the HR schema on the target database Similar to the TABLE parameter the MAP parameter is a complex parameter that has many options that allow you to filter rows map columns transform data and so on Chapter 5 covers some of these options For now let s keep it simple and just instruct GoldenGate to replicate all the changes for the HR schema You can do this easily by using wildcarding with HR which tells GoldenGate to replicate all of the tables in the HR schema You could also list each of the tables individually but using the wildcard is much easier You also assume that your source and target schemas match If they didn t you could use MAP parameter options such as COLMAP to map the different source and target columns www it ebooks info CHAPTER 4 M BASIC REPLICATION Note Notice that the MAP parameter must end in a semicolon Next let s look at adding the Replicat group Adding the Replicat Now that you ve set up the Replicat configuration parameters the next step is to add the Replicat group The following example adds the RHREMDI Replicat to the GoldenGate configuration GGSCI targetserver gt ADD REPLICAT RHREMD1 EXTTRAIL dirdat 12 Let s look at each of the commands in more detail The first command ADD REPLICAT adds the Replicat using the configuration parameters defined in the last section After the R
66. Log out from the terminal session you have open and restart the Oracle GoldenGate processes Oracle GoldenGate Configuration Issues Oracle GoldenGate configuration issues pose challenges because once the environment has been installed and configured if the original DBA has left the company the new DBA team doesn t always have documentation from the past DBA to help them understand the configuration they ve inherited Configuration issues fall into the following areas e Incorrect software version installed for Oracle GoldenGate e Database configuration issues on the source and or target with Oracle GoldenGate 226 www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE e Oracle GoldenGate parameter file configuration issues e Operating system configuration issues e Network configuration issues Incorrect Software Versions with Oracle GoldenGate Installing the wrong version and platform release for Oracle GoldenGate causes failures within the source and target environments Oracle provides a unique build for the GoldenGate software based on platform database and version 32 bit versus 64 bit that requires the correct version to be installed for your platform For instance you can t install 64 bit Oracle GoldenGate Linux software on an IBM AIX platform or you ll receive errors The build name contains the operating system version database version GoldenGate release number and GoldenGate build number as shown
67. Name or IP omms nmm Profile Configuration Run Execute Job Part 7550 Reports Datasource Type Orade Options Settings User Preferences Datasource Connection User Favorites Password sosososo Jobe Edt Datasource connection was successful iah menm d nne Figure 9 2 Connection configuration Setting Up Tables and Data Scripts The following tables and data are used in this tutorial so you can follow this guide You can also use your own tables I have created the table using dba_objects 50 times in the scripts If your system is fast you can add more loops in the code so you can see the Running Jobs option in the left panel create table big_table_source as select 0 seqno a from dba_objects a where 0 1 create table big_table_target as select 0 seqno a from dba_objects a where 0 1 The following script populates the sample tables on the same database If you want to test the comparison on different data you have to copy the data to another database CREATE OR REPLACE PROCEDURE populate_big_table IS BEGIN FOR i IN 1 50 176 www it ebooks info CHAPTER 9 88 ORACLE GOLDENGATE VERIDATA LOOP insert into big source select i a from dba objects a insert into big target select i a from dba objects a commit END LOOP END populate big table Creating a Group In Group Configuration add the source and target connections to the group One Group can have many compare p
68. Now you ve completed part 1 of the switchover process The primary database is no longer being used by the application and the replication from the primary database to the standby database is stopped Now you can proceed to part 2 as shown in Figure 12 4 and start the standby Extract and switch the application to use the standby database 252 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION Start the Extract 1 1 i Local Standby lt Replicat Q Data Pump ES Primary id hi Database 4 eU Maintenance 3 Remote Source Trail Trail Database Application Start the Application 2 Figure 12 5 Disaster recovery planned switchover part 2 Following are descriptions for the steps in part 2 as shown in Figure 12 5 1 To begin capturing changes in preparation for the application switchover alter the Local Extract for the standby database to begin capturing changes now and then start it as follows GGSCI standbyserver gt ALTER EXTRACT LHREMD3 BEGIN NOW GGSCI standbyserver gt START EXTRACT LHREMD3 2 Prepare the standby database for the application and start the application connected to the standby database In preparation you may need to grant some SQL permissions to your application users on the standby database Also remember to enable any triggers or cascade delete constraints you may have stopped while the standby was the replication target 3 At this time you
69. Product Oracle Solaris 10 and 11 Express MySQL NetBeans IDE Oracle GlassFish Server Pre built Developer VMs Java Database SOA amp BPM Enterprise PHP Figure 2 6 Download Oracle Goldengate from OTN website www it ebooks info CHAPTER 2 88 INSTALLATION Once you click the Downloads link from OTN you will be shown the list of Oracle software for download shown in Figure 2 7 E Oracle Software Downloads ORACLE Were ato ian Sa Sto Communitas iama gt E amp Products and Services Downloads Support Edecabos Partners About Oracie Technology Network Berkeiex OB Software Downloads Enitrznse Manaaer Developers Datazase and xe software downloads are Nov and most come with Doyetoper License Mat atows you 10 use fut versions of the products at no charge while develocing and prototyping your applications ox for Pack tee dios strictly sell educalional purposes You can buy products with Kil use censes a ime from the online 25566 or from your gates sence sentative Fusion mogteware sea EE Want to take virtualized test drive See our collection of pre bui Developer VMs for Database App Java and SOADPM development ease Customers and ape you already have commercial license you should download your software from our which is specifically designed for custo
70. RHREMP3 EXTTRAIL dirdat 16 Remember you won t be starting the standby Replicat until a switchover or failover You are now finished with the initial setup for the disaster recovery replication All of the steps you have completed so far are in preparation for a potential switchover or failover situation You are now prepared to perform a planned switchover or an unplanned failover to your standby database Performing a Planned Switchover Even though you are preparing your environment for a complete unplanned disaster recovery scenario there are many situations where you may need to bring down your primary database for a planned outage Using GoldenGate replication you can reduce these planned outage times to the bare minimum of switching your applications from the primary database to the standby database without losing any data during the process You can have your application processing against the standby database while the primary database is down for maintenance and then switch back to the primary database after the maintenance is complete In our example let s assume you need to apply a database software patch to the primary database which normally requires a four hour outage Now let s go through the steps using GoldenGate replication and keep the application outage window as short as possible To keep it simple we ve divided the planned switchover steps into five parts each with several steps Let s begin with Part 1 as shown in Figure 1
71. Remote Source Trail Trail Batebase Applicatian Stop the Application 1 Figure 13 6 Zero downtime migration fallback part 1 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION Following are descriptions for steps 1 4 as shown in Figure 13 6 1 Stop the application running against the new database At this time be sure to leave the Local Extract running so it can capture any remaining database transactions 2 After verifying that there is no lag stop the Local Extract on the new database as shown in the following example GGSCI newserver gt LAG EXTRACT LPREMP2 Sending GETLAG request to EXTRACT LPREMP2 Last record lag 5 seconds At EOF no more records to process GGSCI newserver gt STOP EXTRACT LPREMP2 3 After verifying that there is no lag stop the data pump Extract on the new database GGSCI newserver gt LAG EXTRACT PHREMD2 Sending GETLAG request to EXTRACT PHREMD2 Last record lag 4 seconds At EOF no more records to process GGSCI newserver gt STOP EXTRACT PHREMD2 4 After verifying that there are no more records to process stop the Replicat on the old database GGSCI oldserver gt LAG REPLICAT RHREMD2 Sending GETLAG request to REPLICAT RHREMD2 Last record lag 6 seconds At EOF no more records to process GGSCI oldserver STOP REPLICAT RHREMD2 Now that the replication from the new database to the old database is complete and stopped you c
72. Target Table BIG_TABLE_TARGET Source Column SUBOBJECT_NAME Target Column EDITION_NAME Source Datatype NUMBER Target Datatype VARCHAR2 Comparison Formats Le ing string clob binary Se a Figure 9 6 Change default data type and compare format Using Performance Statistics Veridata keeps a few performance statistics for each run they are in the Finished Job panel as shown in Figure 9 7 You can use these statistics to prepare for the production run for example to calculate the run time for each job and conduct performance tunings 183 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA 184 Compare Pair Performance Start Time Run Duration Rows Processed Rows Per Second Bytes Processed Bytes Per Second Sorting Source 4 23 11 7 01 09 PM 00 00 02 73236 44224 2332796 1408693 Sorting Target 4 23 11 7 01 09 PM 00 00 02 73236 34974 2332796 1114038 Initial Compare 4 23 11 7 01 08 PM 00 00 03 73236 28070 2332796 894134 Confirm Out Of Sync 4 23 11 7 01 09 PM 00 00 02 0 0 0 Compare Pair Performance History O C s Step Sorting Initial Compare Confirm Out Of Sync View Performance History By 50 000 40 000 30 000 20 000 Rows Per Second 10 000 4122 11 10 41 Rows Per Second iw Rows Processed Bytes Processed Bytes Per Second Rows Per Second Run Duration Targ
73. The remote trail has three transactions each with an insert and a delete to the HR EMPLOYEE table The GoldenGate Replicat organizes those transactions into a larger Replicat transaction according to the minimum value of the GROUPTRANSOPS parameter setting For this example let s assume you re using the default GROUPTRANSOPS setting of a minimum of 1 000 SQL operations per Replicat transaction Notice that GoldenGate normally preserves the order of SQL operations with inserts followed by deletes just as in the remote trail Remote Trail Transaction 1 Insert HR EMPLOYEE 1 Update HR EMPLOYEE 1 Delete HR EMPLOYEE 1 Replicat transaction Insert HR EMPLOYEE 1 Update HR EMPLOYEE 1 Delete HR EMPLOYEE 1 Insert HR EMPLOYEE 2 Update HR EMPLOYEE 2 Delete HR EMPLOYEE 2 Insert HR EMPLOYEE 3 Update HR EMPLOYEE 3 Delete HR EMPLOYEE 3 Transaction 2 Insert HR EMPLOYEE 2 Update HR EMPLOYEE 2 Delete HR EMPLOYEE 2 Transaction 3 Insert HR EMPLOYEE 3 Update HR EMPLOYEE 3 Delete HR EMPLOYEE 3 Figure 7 5 Normal GoldenGate transaction processing Figure 7 6 shows GoldenGate processing a remote trail using BATCHSOL The remote trail has three transactions each with an insert and a delete to the HR EMPLOYEE table The GoldenGate replicat organizes those transactions into a larger Replicat transaction according to the GROUPTRANSOPS parameter setting For this example let s assume you re using the default GROU
74. Z zero downtime database upgrades 41 zero downtime migration replication for 263 278 cutover 274 276 fallback 276 278 prerequisites 263 264 requirements 264 265 setup 266 274 topology 265 266 www it ebooks info Expert Oracle GoldenGate Ben Prusinski Steve Phillips Richard Chung Apress www it ebooks info Expert Oracle GoldenGate Copyright 2011 by Ben Prusinski Steve Phillips and Richard Chung All rights reserved No part of this work may be reproduced or transmitted in any form or by any means electronic or mechanical including photocopying recording or by any information storage or retrieval system without the prior written permission of the copyright owner and the publisher ISBN 13 pbk 978 1 4302 3566 8 ISBN 13 electronic 978 1 4302 3567 5 Trademarked names logos and images may appear in this book Rather than use a trademark symbol with every occurrence of a trademarked name logo or image we use the names logos and images only in an editorial fashion and to the benefit of the trademark owner with no intention of infringement of the trademark The use in this publication of trade names trademarks service marks and similar terms even if they are not identified as such is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights President and Publisher Paul Manning Lead Editor Jonathan Gennick Technical Reviewer Arup
75. an example of how you can use encryption for the trail files You can encrypt the source trail file created by the local Extract as shown next Note that the ENCRYPTTRAIL parameter must come before the trail file to be encrypted in this case the 11 trail ENCRYPTTRAIL ExtTrail dirdat 11 Now the 11 source trail file is encrypted The next step is to add encryption parameters to the data pump Extract First you must decrypt the trail file so it can be processed by the data pump and then you can re encrypt it as the 12 remote trail file PassThru DECRYPTTRAIL ENCRYPTTRAIL RmtHost targetserver MgrPort 7809 RmtTrail dirdat 12 Notice that you can still use passthru mode for the data pump even with encrypted trail files The data pump Extract passes the encrypted trail data over the network and writes the encrypted remote trail The last step is for the Replicat to decrypt the trail file so it can be processed as shown in the following example DECRYPTTRAIL Map HR Target HR Let s take another look at your Extract and Replicat parameter files along with the new enhanced security changes introduced in this section shown in italics First is the enhanced Extract parameter file GGSCI sourceserver 1 edit params LHREMD1 Extract LHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default ReportCount Every 30 Minutes Rate Report at 01 00 Repor
76. best be resolved You then saw some specific approaches and examples of tuning GoldenGate by using strategies such as parallel processing groups and using GoldenGate parameters like BATCHSOL and GROUPTRANSOPS www it ebooks info 8 Monitoring Oracle GoldenGate GoldenGate has a lot of moving parts that depend on other systems to operate properly The goal for replication is to replicate data from source to target in real time without losing any data Every system is different and has different strengths and weaknesses You can monitor the weak points but sometimes you may also want to monitor the strong points even though they rarely break As Murphy s Law says Anything that can go wrong will go wrong The cost of fixing a problem may be much more than the expense of implementing a good monitoring system With a good monitoring system you can reduce downtime due to errors or even stop problems before they occur This chapter goes over some scenarios that illustrate what can go wrong in the GoldenGate data propagation chain Next it discusses different ways to detect and identify errors Finally it presents some automated scripts that can make your life easier This chapter teaches you how to identify issues and gives you some basic ideas on how to resolve them To learn about advanced troubleshooting fixing issues and using tuning techniques please refer the chapter 7 11 and 14 of this book Designing a Monito
77. checkpoint time the transaction log sequence number and the relative byte address RBA of the record in the transaction log You can add the DETAIL option to the INFO command to see even more information about the Extract You may find the detail display helpful to see where the important files for the Extract are located such as the parameter and report files The following example uses the INFO command to display details about the LHREMDI Local Extract GGSCI sourceserver 3 info ext LHREMD1 detail EXTRACT LHREMD1 Last Started 2011 01 15 13 53 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoint Oracle Redo Logs 2011 01 17 22 57 50 Seqno 3262 RBA 138240 Target Extract Trails Remote Trail Name Seqno RBA Max MB dirdat 11 3 1489 100 Extract Source Begin End data SourceDB database redo01_01 rdo 2011 01 15 13 53 2011 01 17 22 57 data SourceDB database redo01_01 rdo 2011 01 13 22 29 2011 01 15 13 53 data SourceDB database redo01_01 rdo 2011 01 05 02 13 2011 01 13 22 29 data SourceDB database redo02_01 rdo 2010 12 16 15 28 2011 01 05 02 13 Not Available Initialized 2010 12 16 15 28 64 www it ebooks info CHAPTER 4 M BASIC REPLICATION Current directory gger ggs Report file gger ggs dirrpt LHREMD1 rpt Parameter file gger ggs dirprm LHREMD1 prm Checkpoint file gger ggs dirchk LHREMD1 cpe Process file gger ggs dirpcs LHREMD1 pce Stdout file gger ggs dirout LHREMD1 out Error log gger
78. configuration As a result you have minimized the downtime for your application from four hours to the minimum required to switch your application to the standby database and back Next let s cover a different scenario using our disaster recovery replication to perform an unplanned failover To being capturing changes in preparation for the application switchover back to the primary database alter the Local Extract for the primary database to begin capturing changes beginning now and then start the Extract as follows GGSCI primaryserver gt ALTER EXTRACT LHREMD2 BEGIN NOW GGSCI primaryserver gt START EXTRACT LHREMD2 Prepare the primary database for the application and start the application connected to the primary database In preparation you may need to grant some SQL permissions to your application users Also remember to enable any triggers or cascade delete constraints you may have stopped while the primary database was the replication target Start the data pump Extract on the primary database as shown in the following example GGSCI primaryserver gt START EXTRACT PHREMD2 Start the the Replicat on the standby database as shown in the following example GGSCI standbyserver gt START REPLICAT RHREMD2 Performing an Unplanned Failover In an unplanned failover scenario you lose your entire primary database server to a disaster After the disaster occurs you can use GoldenGate to apply the last database transaction
79. data You can use the INFO command and verify the status of the Local Extract LHREMP2 GGSCI primaryserver 2 gt INFO EXTRACT LHREMP2 If the Extract is not running you can review the GoldenGate error log file and try to resolve the problem You should also run the stats command as shown here for LHREMP2 GGSCI primaryserver 2 gt STATS EXT LHREMP2 You re now finished starting the Local Extract so let s move on to the next step starting the data pump Extract Configuring the Data Pump for Disaster Recovery Now that the Local Extract is started you can proceed with configuring adding and starting the data pump for disaster recovery In our example you configure the data pump to read the 13 trail file written out by the Local Extract named LHREMP2 and pump it over the TCP IP network to the target standby server to trail 14 to be processed by the Replicat One consideration for disaster recovery is you may be sending your trail files over long distances to the remote standby site You should verify you have the network capacity and performance to support your GoldenGate disaster recovery network requirements If needed you can tune the GoldenGate parameters for the network which we reviewed in Chapter 7 From let s edit the parameters for the data pump as shown in this example GGSCI primaryserver 1 edit params PHREMP2 Extract PHREMP2 PassThru ReportCount Every 10000 Records Rate Report at 00 30 246 www
80. data pump to skip reading any data definitions from the database or a definitions file and improves performance Keep in mind the tables being replicated need to be identical to use PASSTHRU Using Parallel Replicats To improve performance you can split the Replicats into parallel processing groups Here are some considerations for splitting your Replicats e Youshouldn t split tables involved in referential integrity into different Replicats e One approach is to use a different Replicat for each schema If the schema is large split it into multiple Replicats e High volume tables can be split into their own Replicats If further separation is needed you can split these high volume tables into separate Replicats based on a key range within the table using the GoldenGate RANGE function For example the Orders table can be split into two Replicats based on the ORDERID key column as shown in the following example In the first Replicat parameter file MAP SCHEMA OWNER ORDERS TARGET SCHEMA OWNER ORDERS AVAILABILITY COLMAP USEDEFAULTS FILTER GRANGE 1 2 In the second Replicat parameter file MAP SCHEMA OWNER ORDERS TARGET SCHEMA OWNER ORDERS AVAILABILITY COLMAP USEDEFAULTS FILTER GRANGE 2 2 Using the Fastest Available Storage Where performance is a concern you should always use the fastest available storage for GoldenGate trail files For best performance store the trail files on dedicated disk in separate mo
81. database versions supported for Oracle GoldenGate 16 replication 22 server xml file 186 Service Level Agreement SLA 180 SET EDITOR command 59 77 SETENV parameter configuring for initial load Replicat file 73 for Local Extract 61 for Replicat process 78 setup 281 283 checkpoint table 282 creating dedicated installation directory 282 creating dedicated users 281 data pump 283 developing naming standards 283 encrypting passwords 282 verifying character sets 283 shell command 221 SLA Service Level Agreement 180 software requirements for Oracle GoldenGate Director 11g 25 26 for Oracle GoldenGate Veridata 29 30 INDEX software versions incorrect 227 sort on database option 180 sort on server option 180 source and target configurations 50 280 source databases 34 35 235 source trails 34 36 SOURCEDEFS parameter 98 SOURCEISTABLE option 71 SOURCEISTABLE task 294 source to target DDL activities 239 source to target DML activities 239 SPECIALRUN parameter 74 SQL DML changes from PAYROLL schema 267 SQL Exec parameter editor 211 SQL permissions 276 SQL Server 2000 Microsoft 112 SQL Server 2005 Microsoft 112 SQL Server 2008 Microsoft 112 SQL Server database Microsoft 118 121 SQL statements 236 SQL gt ALTER DATABASE ADD SUPPLEMENTAL LOG DATA command 55 SQL gt SELECT SUPPLEMENTAL LOG DATA MI N FROM V DATABASE command 54 325 www it ebooks info
82. database systems such as Extract Data Pump Replicat Collector and Manager processes If any of these critical processes fail then replication activities will most likely come to a screeching halt thus impacting data integrity and the operation of real time data transfer between your Oracle GoldenGate environments This section looks first into process failures that occur during normal Oracle GoldenGate operations as well as methods to resolve these types of process failures The preliminary task to perform if you find an Oracle GoldenGate process failure is to investigate the details by using the GoldenGate Software Command Interface GGSCI command INFO ALL www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE C ggs_src gt ggsci Oracle GoldenGate Command Interpreter for Oracle Version 11 1 1 0 0 Build 078 Windows optimized Oracle 11 on Jul 28 2010 17 20 29 Copyright C 1995 2010 Oracle and or its affiliates All rights reserve GGSCI oracledba 1 gt info all Program Status Group Lag Time Since Chkpt MANAGER STOPPED In many cases the Manager process fails when network issues occur on either the source or target environment for Oracle GoldenGate You should check to make sure the Manager process is up and running If the Manager process isn t running then you need to restart it by executing the following command from within GGSCI GGSCI gt START MANAGER To verify that Manager has started successfully
83. detail Tip You can use two dashes at the beginning of a line in the parameter file for comments Keep in mind that GoldenGate has many more configuration parameters than are used in these examples You can refer to the GoldenGate Reference Guidefor a detailed description of all the parameters I usually refer to the Alphabetical Reference section in the GoldenGate Reference Guide to quickly find a parameter its meaning and its various options The following sections also point out any applicable parameter differences for SQL Server EXTRACT You can use the Extract parameter to define the GoldenGate Extract group You should develop naming standards for your Extract group names Having good naming standards will help you identify at a glance what the Extract is used for and where it s running This is especially important in problem solving situations where you need to get downed replication up and running again quickly The current limit for Extract group names is eight characters Here are some items you should include in your naming standard The type of process such as Extract Data Pump or Replicat e The application that is using the process such as human resources payroll sales or billing e The environment where the process is running such as development or production e Anumeric indicator to show if there are multiple Extracts or Replicats In the example the Extract group is named LHREMDI Let s revie
84. diagram as shown in Figure 10 15 for REPGGSt Lag 00 00 00 Figure 10 15 Complate Extract and Relplicat diagram Similar to the initial load mapping add the table in which you want to replicate using the Map icons Finally check your Extract and Replicat parameter files using Param File Editor Make sure your parameter files are similar to Figure 10 16 and Figure 10 17 202 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Param File Editor 1 DIRECTOR2 ACCOUNTB Edit Param Check Figure 10 16 Extract parameter file Param File Editor x Edit Param Check Cancel Figure 10 17 Replicat parameter file In GGSCI type info all to make sure Director has created two groups for you GGSCI gt info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING EXTRACT STOPPED EXTGGS1 00 00 00 01 18 10 REPLICAT STOPPED REPGGS1 00 00 00 01 12 13 When everything is set you can start the extract and replicat from the Director Extract and Replicat icons or GGSCI After you start the Extract and Replicat the Xs on them should be gone The lines between the icons change from grey to green The lag values are also displayed on the icons You can go to the GGSCI tab to check their status as shown in Figure 10 18 Click the Trail icon it also shows the current relative byte address RBA and seqno Feel free to add a new table to the Extract and
85. example a two hour window for cutover and fallback has much different requirements than if an entire weekend is allowed for the cutover Now that you have an understanding of the replication requirements let s review the zero downtime migration topology Zero Downtime Migration Topology The underlying topology for zero downtime migration is one way replication with a few changes for zero downtime migration as shown in Figure 13 1 Chapter 3 Architecture covered the concepts of one way replication One way replication is the simplest topology and is often used for reporting or query offloading purposes In this chapter one way replication topology is used for zero downtime migration purposes Data is replicated from a single source database to a single target database in only one direction at a time Changes to database data are only made at the source database and then replicated to the target database The source database is the old database you re migrating from prior to the migration After the migration cutover to the new database the replication is reversed and data is replicated back from the new database to the old database to allow for migration fallback This backward replication can stay in place for as long as needed Once you re sure you ll stay on the new database and fallback isn t needed you can turn off the backward replication Usually this period is a few days to a few weeks but it can be longer if needed In order t
86. feat impossible with any other tool today In this chapter we will go over the key components of Veridata so you will learn why it is unique and why no other product can compete with it right now Then we will go over some of the key features and benefits and show how Veridata will work with your data projects and help your organization s efficiency Once you understand what Veridata is and how it can help you we will go through a quick tutorial on how to set it up It will literally take only a few minutes to set up a comparison job that is how efficient this tool is Finally we will go over tips and tricks to help you archive some complex data comparison scenarios Veridata Components In order to get Veridata up and running you need to set up five Veridata components as shown in Figure 9 1 www it ebooks info 171 CHAPTER 9 ORACLE GOLDENGATE VERIDATA Oracle Golden Gate Veridata Components Oracle GoldenGate Oracle GoldenGate Veridata Web Veridata CLI Oracle GoldenGate Oracle GoldenGate Oracle GoldenGate Veridata Agent Veridata Server Veridata Agent Source Repositar Target Database y Database Figure 9 1 Oracle GoldenGate Veridata components GoldenGate Veridata Server Veridata Server is the key component of the entire Veridata system Without it the comparison will need to perform on either source or target databases which will negatively impact the performance of the server This is because any compariso
87. in Director Client but it doesn t support the Edit command For a list of nonsupported features in the GGSCI tab in one of the Director dialogs refer to the Director Administrator s Guide On the Monitor Agent tab be sure to turn on monitoring for your data source Modifying the Manager Parameter file Now you can launch the Director Client From the menu bar select File Login enter admin admin if you want to use the default password or enter the login you created in the Oracle GDSC Admin Tool Click the Data Source tab and then drag the data source you created in the Admin Tool to the diagram area Now you should see a Data Source icon as shown in Figure 10 4 Modifying the Parameter File Using the Built in Editor Right click the data source icon in the diagram panel and select Advanced Edit Param File as shown in Figure 10 4 You have full control of the parameter file in this built in editor You ll usually use it to edit parameters that haven t been implemented in the Director GUI 193 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR 194 oldenGate Director 11 1 1 0 0 Figure 10 4 Editing the Manager parameter file Modifying the Parameter File Using the GUI You also can modify the Manager parameter file using the Director interface For example to add the purgeoldextracts parameter to the parameter file highlight the data source and then click the Insert button The dia
88. in Golden Gate extract process Redo logs with sequence 174 and lower are in the archived log area Checking the Archived Log To find out if GoldenGate is reading the online or archived redo log you can use the following GGSCI command 157 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE GGSCI gt Info extract 1 detail EXTRACT EXTGGS1 Last Started 2011 04 09 22 36 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 02 ago Log Read Checkpoint Oracle Redo Logs 2011 04 10 00 16 47 Seqno 175 RBA 36263936 Seqno is 175 because it s the current online redo log If the extract isn t reading the online redo log and behind then the seqno is less than 175 You can check which archived log it s reading as follows SELECT SEQUENCE NAME FIRST CHANGER NEXT CHANGE FROM V ARCHIVED LOG SEQUENCE NAME FIRST CHANGE NEXT CHANGEIt 171 1 8 0000000171 0718807864 0001 6527997 6542114 172 1 8 0000000172 0718807864 0001 6542114 6560323 173 C ARC0000000173_0718807864 0001 6560323 6591047 174 C ARCO000000174_0718807864 0001 6591047 6621440 The default archived log naming convention includes the log sequence If the DBA changed the naming convention you can use sequence to find out which archived redo log file GoldenGate is reading During normal operations max sequence 1 should be the current redo seqno 175 in this case If not the Extract process is behind To determine whether the Extrac
89. info CHAPTER 10 GOLDENGATE DIRECTOR 2 From the drop down menu choose the server name with the TCP port number 3 Click Login The Account page appears 4 the Account page create a new user or use the admin ID 5 Click the Monitor Agent tab and confirm that you re monitoring your data source There are only two roles in Director Client Owner and Observer You can define them on the Data Source tab In addition user activities are logged in the access log file under the Director WebLogic domain Owner has full control of this data source If you leave the Owner field blank then all users can see and access the processes associated with this data source If the Host Is Observable option is selected all non owners are in read only mode for this data source Director stores user specific configurations by user So if you share the same user id you see the exact same screen that other users see such as diagrams and alerts Go ahead and create a data source that has a manager already running In Director terms a data source is also referred to as a Manager process It is the exact same Manager process that Golden Gate uses to manage all the extracts and replicats Note that you must create a Manager parameter file with a required port number in order for the desktop client to work Use Check Connection to confirm if is the manager is running You can t create a Manager parameter file using Director Client You can run GGSCI
90. is working Verifying the Data Pump When the data pump Extract has started you can verify that it s running using the INFO EXTRACT command This is similar to verifying the local Extract as shown in the following example for the PHREMDI data pump Extract GGSCI sourceserver 2 info extract PHREMD1 EXTRACT PHREMD1 Last Started 2011 01 14 10 25 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoint File dirdat 11000001 2011 01 17 15 05 33 485076 977 You should see a status of RUNNING If you see a status of STOPPED or ABENDED there may be a problem As with the Local Extract you can review the GoldenGate error log file and try to resolve the problem Often the error is something simple like incorrect spelling The error file is named ggserr 1og and is located in the GoldenGate software location If your data pump Extract won t start first check for obvious problems such as typographical errors in your parameter file and verify that the source trail file exists in the dirdat directory and that you can access it using the GoldenGate OS user Also make sure the target server is specified correctly in the parameters and verify that you have network connectivity from the source server to the GoldenGate manager port on the target server Chapter 11 goes into more detail about troubleshooting You can see from the INFO command that the data pump status is RUNNING As described earlier checkpoint lag is the
91. issue the Oracle GoldenGate GGSCI command again after you start it The following example makes sure Manager has started correctly and without errors GGSCI oracledba 7 gt start manager Manager started GGSCI oracledba 8 gt info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING As one of the key Oracle GoldenGate processes the Manager process abends or fails if network ports are blocked by a firewall You can check by using the ping command to verify whether there is a network problem with ports You should work closely with your network or system administrator to ensure that the ports allocated by the Manager process are open and available so that Manager can communicate with both the source and target systems for Oracle GoldenGate Oracle GoldenGate Extract Process Failures Oracle GoldenGate Extract process failures can occur either on the source systems or on the intermediate systems that use a data pump Extract process Because the Extract process communicates with the source database Manager process and target Replicat processes if there is a failure with these Oracle GoldenGate processes then Extract either hangs or abends and fail When you re dealing with an Extract failure you should first run the Oracle GoldenGate GGSCI VIEW REPORT command to examine the current status of your Oracle GoldenGate environment If the Extract process fails on the source system run the following command GGSCI VIEW REPORT ex
92. it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION DiscardFile dirrpt PHREMP2 dsc Append DiscardRollover at 02 00 RmtHost standbyserver MgrPort 7809 RmtTrail dirdat 14 Table HR Now that you ve set up the disaster recovery data pump Extract configuration parameters the next step is to add the data pump Extract group on the primary server You can do that using the commands shown in the following example GGSCI primaryserver ADD EXTRACT PHREMP2 EXTTRAILSOURCE dirdat 13 GGSCI primaryserver ADD RMTTRAIL dirdat 14 EXTRACT PHREMP2 MEGABYTES 100 After adding the data pump Extract you can start it to begin processing records from the source trail file as shown in this example GGSCI primaryserver START EXTRACT PHREMP2 Once the data pump Extract has started you can verify it is running using the same INFO EXTRACT command that you used for the Local Extract You should also run the STATS command on the data pump Extract to make sure it is processing changes Now that the Local Extract and data pump Extract for disaster recovery are started you can begin the initial data load You learned about the different methods for initially loading data in Chapter 4 These methods will also work fine for the initial data load for your disaster recovery replication You can use either GoldenGate itself or the DBMS vendor load utilities to do the initial data load Please refer to Chapter 4 for more details on the initia
93. let s add a checkpoint table to the database Adding a Checkpoint Table By default GoldenGate maintains a checkpoint file on disk to keep track of transaction processing In addition you can create a checkpoint table in your target database Doing so lets the Replicat include the checkpoint itself as part of the transaction processing and allows for better recoverability in certain circumstances Let s add a checkpoint table in the target database www it ebooks info CHAPTER 5 ADVANCED FEATURES First you need to add an entry to your GLOBALS file to tell GoldenGate the checkpoint file name By adding this entry to the GLOBALS file you tell GoldenGate to use the same checkpoint table for all of your Replicats You can also add a specific checkpoint table for an individual Replicat as part of the ADD REPLICAT command CheckPointTable gger chkpt Tip The GLOBALS file is used to specify GoldenGate parameters that apply to the entire GoldenGate instance such aS CHECKPOINTTABLE The GLOBALS file is located in the GoldenGate software installation directory in the example gger ggs This is different from the Extract and Replicat parameter files which are located in the dirprm subdirectory If the GLOBALS file doesn t exist you may need to create it the first time you use it Next you create the checkpoint table in the target database GGSCI targetserver 1 gt dblogin userid gger password userpw Successfully logged into datab
94. moving because no more records are sent across the network and written out to the trail file Meanwhile back on the source system the data pump Extract process is unable to move data to the target and soon fails in an abend condition as it runs out of memory for processing data However the primary Extract process group continues to run because it writes out to the local trail file Eventually it reaches the last trail file in the series and when it can t checkpoint it fails and abends This proves the point that it s very important to resolve network issues as soon as possible to prevent the Extract process from falling too far behind in the transaction logs because you don t want data to be out of sync You should monitor the network between the source and target environments by using a network diagnostic tool such as HP OpenView or Tivoli as well as communicating these issues to your network administrator to verify that the network is in a healthy condition and latency issues have been minimized as much as possible After you ve verified that network performance is sufficient check to see if the Extract process bandwidth is saturated You may need to split the processing between multiple Extract groups to remediate network performance issues Multiple data pumps may also benefit environments without large bandwidth resources This reduces the chance of an Extract failing if the network is unreliable and experiences periodic spikes or failure
95. nennen 245 Configuring the Data Pump for Disaster Recovery eeeeeeeere eere nennen nnns 246 Configuring the Replicat for Disaster ReCOVery enses nenne rennen nnn nnns 247 Configuring the Standby 248 Configuring the Standby Data Pump 249 Configuring the Standby Replicat esessssesesesseeseseeeene nennen ennt tntntn 250 Performing a Planned Switchover 2 251 Performing an Unplanned Failover rre 256 SU ee TU 262 Chapter 13 Zero Downtime Migration Replication 263 PREC EC UISILGS m 263 ReguirementS s nen 264 Zero Downtime Migration Topology s sssssssssssssusnsusnsnunnsusnnnnnnnunnnnunununnnnnnnnunnnnnnnnnnnnnne 265 ji 266 Configuring the Local Extract for Zero Downtime Migration eere 267 Configuring the Data Pump for Zero Downtime Migration eere 269 www it ebooks info Configuring the Replicat for Zero Downtime Migration eere rennen 270 Confi
96. off set pagesize 2000 set linesize 250 set trim on set heading off set feedback off spool amp amp SCHEMA add trandata obey select add trandata amp SCHEMA table name from dba tables where owner amp SCHEMA spool off The preceding script generates an Oracle spool file which you can then process in GGSCI using an OBEY command as follows GGSCI sourceserver 1 dblogin userid gger password userpw Successfully logged into database GGSCI sourceserver 2 obey diroby HR add trandata obey Once again it s good practice that when the ADD TRANDATA commands complete you should verify in SQLPLUS that the supplemental logging has been successfully enabled You can do this with a query like the following SOL select owner log group name table name from dba log groups where owner HR OWNER LOG GROUP NAME TABLE NAME HR GGS REGIONS 19375 REGIONS HR GGS COUNTRIES 19377 COUNTRIES www it ebooks info CHAPTER 4 M BASIC REPLICATION HR 065 LOCATIONS 19379 LOCATIONS HR 065 DEPARTMENTS 19382 DEPARTMENTS HR 065 JOBS 19385 JOBS HR 065 EMPLOYEES 19387 EMPLOYEES HR 065 JOB HISTORY 19391 JOB HISTORY For SQL Server 2008 you can query the Change Data Capture tables to determine if changed data capture has been enabled as follows select from cdc change tables The next step is to disable any triggers or cascade delete constraints as discussed in the next section Disabling Triggers and Cascade De
97. on NSK MEGabytes lt nnnn gt For extent size calculation TRANSIND nnn Set the transind field COMMITTS nnn Set the committs field USERTOKEN on OFF detail Show user token info HEADERTOKEN on OFF detail Show header token info GGSTOKEN on OFF detail Show GGS token info FILEHEADER on OFF detail Display file header contents ASCIIHEADER ON off Toggle header charset EBCDICHEADER on OFF Toggle header charset ASCIIDATA ON on Toggle user data charset EBCDICDATA on OFF Toggle user data charset ASCIIDUMP ON off Toggle charset for hex ascii display EBCDICDUMP on OFF Toggle charset for hex ascii display TRAILFORMAT old new Force trail type PRINTMXCOLUMNINFO on OFF Toggle SOL MX columninfo display 298 www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR TMFBEFOREIMAGE on OFF Toggle display of TMF before images FLOAT lt value gt Interpret a floating point number FORMAT lt specifier gt sprintf format default f Logdump 8 gt HISTORY The Logdump HISTORY command shows all recent activity performed in a Logdump session as shown in the following example Logdump 8 gt history 1 gt ghr on 2 gt help 3 gt open dirdata aa 4 gt open aa 5 open c ggs src trails aa 6 gt host 7 gt help 8 gt history Opening GoldenGate Trail Files with Logdump The most common use for Logdump in Oracl
98. online Oracle GoldenGate documentation guides available at http otn oracle com Sybase to Oracle Replication Sybase was originally a joint venture between Microsoft and Sybase Corporation before SQL Server split off from Sybase and Microsoft to become two different database products Sybase has two database products Sybase IQ and Sybase SQL Server Currently there is no support for Sybase IQ to replicate with Oracle GoldenGate However replication from Sybase SQL Server is supported with Oracle GoldenGate Table 6 4 provides the currently supported versions of Sybase with Oracle GoldenGate Table 6 4 Supported Configurations for Oracle 11g GoldenGate with Sybase Source Source OS Target Target OS GG Version Sybase Microsoft Windows Linux Oracle 9i 10g Windows Linux Oracle 11g 12 x UNIX 115 UNIX GoldenGate Sybase 15 Microsoft Windows Linux Oracle 9i 10g Windows Linux Oracle 11g UNIX llg UNIX GoldenGate Details on how to configure Sybase to Oracle replication can be found in the online Oracle GoldenGate documentation guides available at http download oracle com docs cd E18101 01 doc 1111 e17806 pdf 123 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION IBM DB2 UDB to Oracle Replication IBM DB2 UDB provides an enterprise data warehouse and OLTP Online Transaction Processing database platform for Windows UNIX Linux and mainframe platforms With the advent of Oracle Fusion Middleware popularity a
99. other activity such as rolled back changes Extract can also be configured to write the Trail File directly to a remote target server but this usually isn t the optimum configuration In addition to database data manipulation language DML data you can also capture data definition language DDL changes and sequences using Extract if properly configured You can use Extract to capture data to initially load the target tables but this is typically done using DBMS utilities such as export import or Data Pump for Oracle You can configure Extract as a single process or multiple parallel processes depending on your requirements Each Extract process can act independently on different tables For example a single Extract can capture all the changes for of the tables in a schema or you can create multiple Extracts and divide the tables among the Extracts In some cases you may need to create multiple parallel Extract processes to improve performance although this usually isn t necessary You can stop and start each Extract process independently You can set up Extract to capture an entire schema using wildcarding a single table or a subset of rows or columns for a table In addition you can transform and filter captured data using the Extract to only extract data meeting certain criteria For example you may want to filter a Customer table to only extract customer data if the customer s name is equal to Jones 35 www it ebooks info
100. providing a way to implement the new upgraded database platform ahead of time and have it ready and waiting for the application to cutover The only downtime needed is the time for the application to cutover and reconnect to the new target database Also by replicating from the new upgraded target back to the old source database Oracle GoldenGate can provide a way to easily fall back to the old database if there are problems with the migration In addition Oracle GoldenGate has the advantage of working across many different heterogeneous databases and hardware platforms For example you can use Oracle GoldenGate to migrate from NonStop SQL MX to the latest release of an Oracle database on Red Hat Linux Using Oracle GoldenGate you can develop standard zero downtime database migration processes and implement them across your organization regardless of the specific DBMS or platform being used This chapter introduces another application of Oracle GoldenGate replication to implement a specific replication configuration for zero downtime migration purposes This chapter covers setting up and configuring Oracle GoldenGate replication for zero downtime migrations It covers how to handle both cutover to the new database and fallback from the new database to the old database if there are problems This chapter refers to the database you re migrating from as the old database and the database you re migrating to as the new database Prerequisites You nee
101. s review a few DBMS configuration options that are specific to those of you using an Oracle database in your replication configuration First you see how to configure GoldenGate in an Oracle Real Application Clusters RAC environment Then you review some special configuration options for Oracle Automatic Storage Management ASM Finally you see how to set up DDL replication between Oracle databases If you aren t using Oracle databases you can skip this section www it ebooks info 99 CHAPTER 5 ADVANCED FEATURES Configuring for Oracle RAC GoldenGate has had support for Oracle RAC for many years There are some special considerations you should be aware of if you re running GoldenGate in Oracle RAC Consult the Oracle GoldenGate Oracle Installation and Setup Guide for specific platform requirements You can choose to install and run GoldenGate from any of the nodes in your RAC cluster You may decide to run GoldenGate on one of the nodes that is planned to have less workload than another node GoldenGate can be installed on shared storage that is available to all the nodes in the RAC cluster or on local storage If you install GoldenGate on shared storage you can start the GoldenGate Manager and other GoldenGate processes from any of the nodes If the node on which you started GoldenGate crashes you can restart it on another node You can also set up GoldenGate to run under Oracle Clusterware and automatically fail over to ano
102. s take a look at the parameters for the Local Extract on the primary server as shown in the following example Since we used LHREMDI in an earlier chapter let s name the disaster recovery Local Extract LHREMP2 GGSCI primaryserver 1 edit params LHREMP2 Extract LHREMP2 USERID GGER PASSWORD userpw ExtTrail dirdat 13 ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt LHREMP2 dsc Append DiscardRollover at 02 00 Table HR 245 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION These parameters should look familiar as we covered them all in Chapters 4 and 5 The next step is to add the Extract to the GoldenGate configuration You should pick a new unused trail file name In this example you can use trail file 13 for the Local Extract trail since trail files 11 and 12 were used in previous examples You can add the extract using the following commands from GGSCI GGSCI primaryserver gt ADD EXTRACT LHREMP2 TRANLOG BEGIN NOW GGSCI primaryserver gt ADD EXTTRAIL dirdat 13 EXTRACT LHREMP2 MEGABYTES 100 After adding the Extract you need to start it to begin capturing changes as shown in the following example GGSCI primaryserver gt START EXTRACT LHREMP2 You should make sure the Extract is running properly after it is started If you see a status of STOPPED or ABENDED there may be a problem You should see a status of RUNNING and the RBA value increasing if you are processing
103. security check1 OracleGoldenGate com Change Password Help Log Out ORACLE GoldenGate Network 2 Processes Tasks _ Select UnselectAl Stop jpStat PKU 2 Entire Network Manager Process Type Status tag Data Source Show detalis w i Jocainost_ga localhost gg WU ORA S Active Watch localhost go EXTGGS1 138 REPINIT J eg ExTGGS Capture Running 00 00 00 ORCLREDO02 LOG Redo Log 18 ExTGGS1 More Info Last Record 4 2 11 11 02 42 AM Last Status 4 2 11 11 03 01 GGSCI eese 1 4 REPGGS1 gt amp Watch Lists lt My Email Alerts Live Monitoring is ON Event Log Select All Unselect All Filter gg Acknowiedge 2 CJ G NFO localhost og EXTGGS1 CAPTURE 411 10 03 21 PM 06501517 Oracle GoidenGate Capture for Oracie EXTGGS1 orm Position of first record processed Sequence 137 RBA 16522768 SCN 0 5622702 Apr 1 2011 8 38 08 PM D INFO localhost og EXTGGS1 CAPTURE 411 10 03 21 PM 06601478 Oracie GeidenGate Capture for Cracie EXTGGS1 prm Output file C ggstdirdaliC is using format RELEASE 10 4 11 1 eJ 1 INFO localhost og EXTGGS1 CAPTURE 411 10 03 21 PM 06601052 Oracle GoldenGate Capture for Oracle EXTGGS1 prm No recovery is required for target file C ggsidirdatiC 1000000 at RBA 0 fie not opened O wro localnost_gg EXTGGS1 CAPTURE 4 1 14 10 03 20 PM 06600993 GeidenGate Capture for Cracie EXTGGS1 prm EXTRACT EXTG
104. separate server So it removes the overhead from the source and target databases It is non intrusive so it enables real time data comparisons with zero downtime on the changing data We can compare the data between two OLTP systems while their data is changing Veridata will use two phase confirmation on the data synchronizations Supports data or database migration projects in both initial copy and real time sync phases to ensure the data accuracy We can use GoldenGate to do no downtime migration but with Veridata running on the background to cross check the data between the source and migrated databases we don t need to write expensive queries to do the data comparison during any phases of the migrations Supports cross database comparison between Oracle SQL Server Teradata HP Enscribe and HP SQL MP databases Easy to use It can be used by both IT and business users We can set up most of the compares without writing a single SQL statement It is a standalone tool that is not dependent on Oracle GoldenGate You don t need to install GoldenGate at all to run Veridata It has role based security to control access and protect sensitive data Selective comparison compares selected columns It can also do incremental compares with appropriate setup It has a flexible command line interface to automate and schedule compare jobs www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA Setting Up the Veridata Compares
105. should perform any maintenance required on the primary database while there are no applications connected In the example we are going to bring down the primary database and apply a database patch After completing part 2 you should be done with your database maintenance and ready to switch the databases back to their original configuration In order to do that you must apply any changes made to the standby database while it was functioning as the primary database back to the original primary database Those steps are shown in Figure 12 6 253 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION Start the Replicat Start the Pump 3 4 pa Local Stendby Ei Replicat ES aa Data Pump AE Primary 4 Database Finish Maintenance 1 Remote Betebase Applieation Stop the Application 2 Figure 12 6 Disaster recovery planned switchover part 3 Now let s review each of the steps in part 3 as shown in Figure 12 6 1 Firstyou need to complete any maintenance activities on the primary database Until the maintenance is successfully completed the standby database will act as the primary Make sure to disable any triggers or cascade delete constraints on the primary before processing the GoldenGate changes from the standby 2 Stop the application running against the standby database At this time be sure to leave the Local Extract running so it can capture any remaining data
106. snd changed bytes or values to be used by GoldenGste Compressing updates gives GoldenGste sll of the datas required to update the target record while restricting the amount of data thst must Figure 10 25 Nocompressupdates is in the drop down list instead of the parameter list panel As you can see in Figure 10 25 the compressdeletes parameter isn t in the list neither is nocompressdeletes These are important parameters if you need to get full values of the deleted rows in the trail file You have to add them using the Edit Param feature Advanced Mapping Select the Map for EXTGGSI icon in the diagram and then click the Change button on the parameter tab or Insert if you need to add a new mapping the dialog shown in Figure 10 26 opens Note that if you change the Mapping Type in the Change Mapping dialog all attributes will disappear from the dialog they re still in the parameters file but you have to reopen this dialog to get them to reappear www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Change Mapping GG_DIRECTOR2 ggs_test MyFunction id get_userid params parami userid DBUSERID A ORAENVIRONMENT DBLOGINUSER SYSDATE GETENV 27 TABLE TABLE is synonym for the MAP psrameter One exception for TABLE the TARGET psrameter is optional Figure 10 26 Mapping Type has a lot of attributes 211 CHAPTER 10 GOLDENGATE DIRECTOR Figure 10 27 shows
107. software must be installed prior to deploying Oracle GoldenGate Director e Java Runtime Environment RE release 6 1 6 or later required on the system that will house Oracle GoldenGate Director Server e Oracle Weblogic Server 11g 10 3 1 10 3 2 10 3 3 Standard Edition Server 25 www it ebooks info CHAPTER 2 INSTALLATION Note Oracle 11g Weblogic Server must be installed and configured prior to performing the Oracle 119 GoldenGate Director Server installation Please consult the Oracle 11g Weblogic documentation for details on how to install Oracle 11g Weblogic In addition to the above software requirements you will also need to install a new repository database for the Oracle 11g GoldenGate Director Server during the installation process The following RDBMS database platforms are supported as the repository database for Oracle GoldenGate Director Server e Oracle 91 or later e MSSQL Server 2000 or 2005 e MySQL 5 x Enterprise Edition For UNIX and LINUX platforms a graphical GUI windowing environment such as X Windows must be installed and configured GoldenGate Director Client Installation Requirements Before you install the client for Oracle GoldenGate Director be advised that you will need to complete the following preparations e The monitor screen display resolution must be at a level of 1024 x 768 or higher on the client machine e JRE version 6 software for the Java Runtime Environment must be install
108. solid understanding of how to tune GoldenGate The process of tuning GoldenGate is similar to many software tuning exercises however the tuning tools and mechanisms are quite different You should also keep in mind that although GoldenGate has its own tuning approaches its performance depends on the performance of the underlying database server network and storage This chapter gives you an in depth understanding of how to tune GoldenGate replication This chapter reviews a structured methodology for tuning GoldenGate replication and digs into each step of the methodology You then focus on some specific tuning strategies and examples you can use to improve the performance of your GoldenGate replication environment Tuning Methodology When tuning GoldenGate it s important for you to follow a standard performance and tuning methodology Otherwise you may not know when there is a performance problem or you may find yourself fixing the wrong problem This chapter suggests a standard methodology but it s not important that you follow it precisely You can create your own methodology or adapt this methodology to work in your particular environment The most important consideration is that you do follow a structured approach that has requirements defined up front and measurements and verification specified afterward Using a structured approach helps ensure that you can respond properly to performance issues and meet the expectations of your use
109. t tell you exactly what went wrong but the other scripts in this chapter can pinpoint for you the exact issues This chapter also showed you how to monitor the health of your overall GoldenGate systems Before you implement a GoldenGate monitoring process it s important to know what can go wrong in your systems so you can identify problems before they get worse Note The Management Pack for GoldenGate also called GoldenGate Director can do some basic monitoring and alerts Director also stores the lags and events information to its proprietary tables so you can create reports from them But use it at your own risk because Oracle may change its table structures from version to version See Chapter 10 for more information about Director 169 www it ebooks info 9 Oracle GoldenGate Veridata GoldenGate Veridata is a high performance cross platform data comparison tool that supports high volume compares while both source and target data are online and the data is being updated Veridata three tiered architecture spreads out the loads among different servers without compromising the performance of the source and target database systems Veridata uses compressed row hash data to do a comparison that reduces the CPU cycles and network bandwidth required for comparing the data but it s not merely a data comparison tool Veridata can keep track of changing data and perform comparisons when the data is being updated a
110. target Microsoft SQL Server environment as well as the GGS user security account Make sure to assign the correct level of system and database privileges to the system and database account 119 www it ebooks info CHAPTER 6 HETEROGENEOUS REPLICATION Since the Oracle GoldenGate Extract and Replicat process groups both connect to a Microsoft SQL Server database via an ODBC Open Database Connectivity connection you will need to set up these Oracle GoldenGate components with a system data source name DSN To do so perform the following tasks on the Microsoft SQL Server host 1 2 3 5 120 Under Programs select Start gt Settings gt Control Panel Choose gt Administrative Tools Open the ODBC Data Source Administrator dialog box by double clicking the option for Data Sources ODBC Next be sure to click the System DSN tab and then hit the Add button The Create New Data Source dialog box will then appear as shown in Figure 6 7 Create New Data Source x Select a driver for which you want to set up a data source Microsoft Paradox Driver db Microsoft Paradox Treiber db Microsoft Text Driver txt csv Microsoft Text Treiber txt csv Microsoft Visual FoxPro Driver Microsoft Visual FoxPro Treiber Oracle in 92 WN CO d d d ata Figure 6 7 Create New Data Source for the Microsoft SQL Server database Create the new data source on t
111. than 2MB in length Any DDL longer than that must be processed manually using a GoldenGate provided script DDL replication is supported for replication between only two systems Bidirectional DDL replication is supported The source and target schemas must be identical This means you must be using the ASSUMETARGETDEFS parameter on your Replicat You must use passthru mode on your data pump Extracts for tables that require DDL replication You should keep the DML and DDL replication for a single table and for related groups of tables together in the same Extract and Replicat groups Consult the Oracle GoldenGate Windows and UNIX Administrator s Guide for details on additional restrictions Now let s go though the steps to add Oracle DDL replication This example uses the gger user schema to store the DDL tables but you can choose another schema if needed 1 Grant execute permission on UTL_FILE to the gger user SOL grant execute on utl file to gger Add the following line to your GLOBALS file Remember this file is in your GoldenGate software installation directory If it doesn t exist you need to create it the first time GGSCHEMA GGER Run the marker 591 script from the GoldenGate software installation directory as shown next This script is included as part of the GoldenGate software installation It prompts for the schema name which in this case is gger 103 www it ebooks info CHAPTER 5 ADV
112. the GoldenGate Director Director Server database that manages all the jobs Director Web that is used to do some simple monitoring and administrations and lastly Desktop Client that is used to do all the Director tasks 189 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Oracle GoldenGate Development Administrator Oracle Weblogic Server GGSCI Data GGSCI Oracle GoldenGate Oracle GoldenGate Director Server Domain Warehouse Active DB 1 Director Web Oracle GoldenGate Oracle GoldenGate Director Server Director Desktop Client Monitor GGSCI ODS GGSCI Agent Passive DB GGSCI GGSCI Active DB 2 Standby Oracle GoldenGate Director Database Figure 10 1 Director components GoldenGate GGSCI Instances GoldenGate director can link to many Manager processes or referred data source in the Director Client application The Manager IP address and port number are the only required parameters required for Director Server to connect to the GGSCI instances Director Administrator Director Administrator is used to set up all the GGSCI instance configurations All of this information is stored in the Director repository database Director Server and Database Director Server runs as a WebLogic domain There is no need to run the default WebLogic domain when you start up Director Server However you need to install the WebLogic server components Director Monitor Agent is part of Director Server so you don t nee
113. the Maximum Threshold It simportant to get the maximum threshold for how much GoldenGate can transfer through the network For example if GoldenGate has to be down for 5 hours due to a scheduled outage you need to know how long it will take GoldenGate to catch up to zero lag To do this you can stop the data pump for few hours and then restart it You can use the stats command at the GoldenGate Software Command Interface GGSCI prompt to calculate the number of operations In addition you can check the accumulated file size and consumption rates You see how to do that later in this chapter Which Processes to Monitor in the GoldenGate Environment This chapter uses Extract EXTGGSI Data Pump DPGGSI and Replicat REPGGSI for demonstration purposes It s a simple one way extract GoldenGate processes use CSN seqno and RBA to connect and sequence each process together CSN seqno and RBA are the tools to identify the lag in GoldenGate Environments If you know exactly where the lags are then you can put more monitoring in that process As mentioned earlier Oracle GoldenGate uses CSNs to track the commit order Each database system has its own unique serial number refer to the Oracle GoldenGate Administration Guide for a complete list of CSNs The CSN is also associated with the timestamp Note In a Real Application Clusters RAC environment it s very important to have all the clustered servers in the exact same timestamp Otherwise yo
114. the Veridata schema account to be used in the installation of the Veridata Server For instance if you choose to use MySQL for the repository with Veridata Server you will need to first create a user and database and then grant DDL and DML privileges to this account With Oracle as the repository database for Veridata you need to grant database privileges to the newly created Veridata repository schema account GRANT the following privileges to the VERIDATA ROLE e CREATE TABLE e CREATE VIEW e CREATE SESSION e CREATE PROCEDURE In addition you will need to grant the VERIDATA_ROLE to the user for Oracle as well as grant the QUOTA UNLIMITED privilege to the Veridata user as default tablespace If you should choose to use Microsoft SQL Server for the Veridata Server repository database platform you will need to create a new database and login for the Veridata user Furthermore the following privileges will need to be granted e ALTER SCHEMA e CREATE DROP TABLE ALTER e CONNECT e CREATE INDEX e DROP INDEX e INSERT UPDATE DELETE e SELECT GoldenGate Veridata Web Requirements for Installation For the Oracle GoldenGate Veridata web requirements you will need to ensure that one of the following Internet browsers is available on the client web host e Mozilla Firefox 1 0 4 or later version 30 www it ebooks info CHAPTER 2 88 INSTALLATION e Microsoft Internet Explorer 6 or later version Oracle GoldenGate Veridata mak
115. the data pump You can add the data pump Extract using the commands shown in the following example GGSCI standbyserver gt ADD EXTRACT PHREMP3 EXTTRAILSOURCE dirdat 15 GGSCI standbyserver ADD RMTTRAIL dirdat 16 EXTRACT PHREMP3 MEGABYTES 100 You won t be starting the data pump until later so let s move on to configuring the standby Replicat Configuring the Standby Replicat Next you should add a standby Replicat which will be ready to apply any changes back to your primary database after a failover or switchover This is similar to the primary replicat but will be used to replicate back to the primary database from the standby database By adding it now before it is needed you will be ready to start it in the event of an outage First create a parameter file for Replicat RHREMP3 as shown in the following example GGSCI primaryserver 1 edit params RHREMP3 Replicat RHREMP3 USERID GGER PASSWORD userpw 250 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt RHREMP3 dsc Append DiscardRollover at 02 00 HandleCollisions should be turned off after the initial load synchronization HandleCollisions AssumeTargetDefs Map HR Target HR The next step is to add the standby Replicat group In the following example you are adding the RHREMPS replicat to the GoldenGate configuration GGSCI primaryserver ADD REPLICAT
116. the new changes to take effect The following example shows how to stop the Replicat RHREMDI GGSCI targetserver gt STOP REPLICAT RHREMD1 The next section looks at verifying the Replicat 81 www it ebooks info CHAPTER 4 M BASIC REPLICATION Verifying the Replicat When the Replicat has started you can verify that it s running using the INFO REPLICAT command You should see a status of RUNNING If you see a status of STOPPED or ABENDED there may be a problem You can review the GoldenGate error log file and try to resolve the problem Often the error is something simple like incorrect spelling The error file is named ggserr log and is located in the GoldenGate software installation location In the example the error log on Linux is gger ggs ggserr log On Windows the GoldenGate error log based on the example is located in c gger ggs ggserr log If your Replicat won t start first check for obvious problems such as the wrong GoldenGate database user ID and password or typographical errors in your parameter file You should also verify that the target database is online and that you can connect to it using the GoldenGate database user Chapter 11 goes more into more detail on troubleshooting Let s do an INFO command on the Replicat RHREMDI to check the status GGSCI targetserver 2 info replicat RHREMD1 REPLICAT RHREMD1 Last Started 2011 01 22 22 40 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 09 ago Log Re
117. this time be sure to leave the Local Extract running on the standby database so it can capture any remaining database transactions 4 Start the Replicat on the primary database to prepare to process the changes that were made on the standby database as shown in the following example GGSCI primaryserver gt START REPLICAT RHREMD3 5 Start the data pump Extract to send database change transactions to be processed by the Replicat Once the replication is complete and the lag is showing At EOF no more records to process you can stop the Extracts and Replicats as shown in the following example GGSCI standbyserver gt START EXTRACT PHREMD3 GGSCI primaryserver gt LAG REPLICAT RHREMD3 Sending GETLAG request to REPLICAT RHREMD3 Last record lag 6 seconds At EOF no more records to process GGSCI primaryserver gt STOP REPLICAT RHREMD3 260 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION GGSCI standbyserver gt STOP EXTRACT LHREMD3 GGSCI standbyserver gt STOP EXTRACT PHREMD3 Note Once you have finished synchronizing the primary and standby databases you should perform data validation You can do this manually with sql queries for row counts and selected data values or use an automated tool such as Oracle GoldenGate Veridata Refer to Chapter 9 for more information on Veridata The last part is to move the users and the application back to the primary database and restart the ori
118. to begin processing changes from the data pumps GGSCI targetserver start rep RHREMD Sending START request to MANAGER REPLICAT RHREMD1 starting Sending START request to MANAGER REPLICAT RHREMD2 starting GGSCI targetserver info rep RHREMD www it ebooks info CHAPTER 7 88 TUNING REPLICAT RHREMD1 Last Started 2011 03 20 22 53 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 04 ago Log Read Checkpoint File dirdat 12000016 2011 03 20 22 51 22 493945 RBA 1559 REPLICAT RHREMD2 Last Started 2011 03 20 22 53 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 04 ago Log Read Checkpoint File dirdat 13000000 First Record 0 11 After some changes are applied you can again check the status of the Replicats to make sure they re processing properly You can check the RBA and make sure it s increasing You can also doa stats replicat tablename command andastats extract tablename command to make sure the Extracts and Replicats are processing changes GGSCI targetserver info rep REPLICAT RHREMD1 Last Started 2011 03 20 22 53 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 00 ago Log Read Checkpoint File dirdat 12000016 2011 03 20 22 56 37 000067 RBA 1856 REPLICAT RHREMD2 Last Started 2011 03 20 23 07 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoint File dirdat 13000000 2011 03 20 23 00 05 000105 RBA 1086 Next let s look at another strategy for creatin
119. to run successfully Missing Oracle GoldenGate Process Groups One common issue that plagues Oracle GoldenGate environments is missing process groups As a starting point in troubleshooting such issues you should execute the GGSCI INFO ALL command to view all of the current processes and groups on the system The Extract group name may have been spelled incorrectly either when the group was created or when the START command was issued for the Oracle GoldenGate process Missing Oracle GoldenGate Trail Files Often Oracle GoldenGate environments have Extract parameter files or obey files that create Extract process groups that reference non existent or missing trail files Without a valid trail file an Extract can t write out data captured from the online redo log files in the source database Furthermore the Replicat process will abend and fail because it can t read from the remote trail file Without trail files the Extract can t write the initial checkpoint and the Replicat has no available data source to read You can issue the GGSCI command INFO EXTRACT lt group gt or the INFO REPLICAT lt group gt command with the DETAIL option to verify whether the trail file exists In addition after you identify the trail file you can use the GGSCI command INFO EXTTRAIL lt trail file name gt to drill down further GGSCI oracledba 3 gt info extract extora detail EXTRACT EXTORA Initialized 2011 03 13 23 45 Status STOPPED Checkpoint Lag 00 0
120. to wait until the initial load is completely finished before starting the Replicat for ongoing change synchronization Later this chapter reviews starting the ongoing synchronization Replicat First let s review the steps for loading with DBMS vendor utilities 75 www it ebooks info CHAPTER 4 M BASIC REPLICATION Loading with DBMS Utilities The DBMS vendor load utilities are optimized to load the particular vendor database so you may find that they perform better than the GoldenGate load For example for an Oracle database you can choose from the original export import the Data Pump export import RMAN copy transportable tablespaces and so on You should choose the load utility that works best for your particular needs For example if you re using GoldenGate to replicate from Oracle 91 to Oracle 11g the original export import may be your best option for the initial load If you re replicating from Oracle 10g to Oracle 11g then you may decide to use Oracle Data Pump export import You can refer to the specific DBMS manuals for requirements and instructions on running the load utilities Prerequisites for Loading with DBMS Utilities Before starting the initial load using DBMS utilities make sure the following tasks have been completed e You should disable any foreign key constraints and triggers on the target tables to prevent data errors Check constraints can be disabled during the load to speed up performance Remember if you
121. undo tablespace on the source and target Oracle databases Summary This chapter provided you with tips and techniques for how to troubleshoot Oracle GoldenGate issues from a holistic perspective You learned how to identify and resolve common issues that cause failures with Oracle GoldenGate processing due to database operating system and network configuration setups In addition the chapter discussed ways to enable tracing of the Oracle GoldenGate environment to isolate the root cause of operational issues For more information see My Oracle Support at http support oracle com www it ebooks info 12 Disaster Recovery Replication There are many strategies available for maintaining a disaster recovery database These range from simply keeping backup copies of the database at an offsite location to database level replication and even storage mirroring This chapter will give you a good understanding of how to set up up the GoldenGate replication solution for maintaining a disaster recovery database The major advantage of using GoldenGate for disaster recovery replication is that it can become your organization s universal database disaster recovery solution GoldenGate works across many different databases and platforms Using GoldenGate you can develop standard disaster recovery processes and implement them across your organization regardless of the specific DBMS being used Earlier in Chapter 4 you set up basic rep
122. use GoldenGate for the initial load e The application that is using the process such as human resources payroll sales or billing e The environment where the process is running such as development or production e An indicator to show if there are multiple Replicats In the example the Replicat group is named DHREMDI The first character D specifies that it s the direct load Replicat HREM specifies that you re using this Replicat group for the human resources employee application and the HR schema The second D signifies that you re running in the development environment Any finally the 1 indicates that this is the first Replicat group for this schema and application You can add additional Replicat groups to improve performance as discussed further in Chapter 7 www it ebooks info CHAPTER 4 M BASIC REPLICATION SETENV You can use the SETENV parameter to override default environment variable settings The example explicitly sets NLS LANG to AMERICAN AMERICA AL32UTF8 It s a good idea to set NLS_LANG and not let it default to the environment setting If NLS LANG is incorrect it can cause your replication to fail or replicate incorrect data For the Replicat NLS_LANG needs to match the NLS character set of the data in the trail file it s reading which was created from the source database You can set other environment variables in addition to NLS_LANG using the SETENV parameter For example you can set ORACLE HOME and
123. using a 64 bit operating system you will be able to take advantage of additional memory for virtual memory requirements should you perform large sorting operations with Veridata GoldenGate Veridata Server Software Requirements Repository database is installed on the Oracle GoldenGate Veridata Server host The following database platforms are supported for the Veridata Server repository e Oracle e MySQL e MSSQL Server 29 www it ebooks info CHAPTER 2 INSTALLATION The Veridata installer will create the database for you during the installation process However you will need to have staged the database repository software prior to running the installer For instance if you choose to use Oracle as the repository for the Veridata server you will need to make sure that the client and server software for Oracle is available and staged on the Veridata Server host before you begin the installation process Furthermore for Oracle you will need to use either the TNSNAMES or EZCONNECT method for the database network services All of the correct values MUST be provided for the Oracle network services such as the database instance name in the tnsnames ora and listener ora file if you choose to use the TNSNAMES with the Oracle repository database for Veridata Database Privileges for GoldenGate Veridata Server As part of the pre installation tasks for the Oracle GoldenGate Veridata Server you must grant the appropriate database privileges to
124. was created by the local Extract or by the data pump Extract The second character can be any valid letter or number For example trail P1 can be used for a data pump trail Using a Data Pump Although technically a data pump Extract isn t necessary you should almost always use a data pump The local Extract can be configured to send changes directly from the source server to the remote target without a data pump Adding a data pump to the configuration however introduces another protective layer to insulate the local Extract from any disruptions caused by the network connection to the target or from a problem with the target itself For example if there was a network issue between the source and the target this could cause the local Extract to fail By having a data pump between the source and the target the local Extract can continue to extract changes and only the data pump is impacted This way when the network issue is resolved you can restart the data pump and quickly process the queued changes that the local Extract has already captured in the trail Keep in mind that the data pump requires an extra trail file so you should make sure to allow some extra disk space in your configuration 283 www it ebooks info CHAPTER 14 TIPS AND TRICKS Management and Monitoring This section covers some tips and tricks that make it easier to manage and monitor your GoldenGate environment By utilizing these suggestions you can simplify th
125. when Manager starts Another parameter AUTORESTART is also available but should be used with caution Sometimes a process fails due to a temporary resource issue or network problem and simply needs to be restarted AUTORESTART automatically attempts to restart any failed processes To be sure you are not unknowingly overlooking any problems you may prefer to have an Extract or Replicat process fail and then resolve the issue manually instead of using AUTORESTART Performance This section covers some tips and tricks for making your GoldenGate setup achieve the best possible performance Following these suggestions early in your project will help ensure that you don t experience performance issues down the road Running Performance Tests You should always test the performance of your GoldenGate setup by running tests that execute or simulate the execution of your actual production processing workload This is the single best insurance against future problems Ideally the tests should be run on a system with hardware and software identical to the production system In addition you should use a copy of the actual production data or at least a realistic subset of the production data for testing against a typical production workload During the tests you can capture statistics about GoldenGate performance and measure the lag to determine if the configuration can handle the load You can also gather server network and database statistics and revi
126. you re doing mapping on the source using the Extract you create a data definitions file on the target server and transfer it to the source server Tip GoldenGate doesn t consider tables identical if any of the columns don t match or are in a different order The names lengths data types semantics and column order must match exactly for GoldenGate to consider the tables identical 97 www it ebooks info CHAPTER 5 ADVANCED FEATURES 98 The first step in generating a data definitions file is to create a defgen parameter file to tell GoldenGate what needs to be included in the file The following example creates a defgen parameter file for all the source tables in the HR schema In this case the defgen file has the same name as the Replicat but with a different file name extension defs defsfile dirdef RHREMD1 defs USERID GGER sourcedb PASSWORD userpw TABLE HR Next you can generate the data definitions file using the GoldenGate defgen command defgen paramfile dirprm hrdefs prm This generates the data definitions file RHREMD1 defs as specified in the defgen parameter file The next step is to transfer this file from the source server to the target server You should put the defgen file in the dirdef directory The last step is to tell GoldenGate that you re using a defgen file and no longer using the ASSUMETARGETDEFS parameter because the source and target tables are different In the following example the ASSUMET
127. you view the unstructured trail data in hex and ASCII format and count the number of records in the trail for debugging purposes You can also filter the trail for desired records and save them to a new trail for reprocessing if needed Reverse If you ever need to undo database changes you can use the GoldenGate Reverse utility Reverse uses a special Extract to capture the before images of changes and then reverses the order of the transactions and writes a new trail file for doing the reversal A Replicat then processes the special reversed trail to undo database changes Reverse can be useful for backing out unwanted changes to the database or quickly restoring a test database back to its original state Reverse is another utility that is included with the GoldenGate software installation Veridata Veridata is an additional product from GoldenGate that you can use to verify that your source and target databases match For critical business applications you should always verify that your source and target databases match when doing replication If the databases being replicated are used for failover this is especially important Data verification is usually time consuming and difficult to accomplish for large amounts of data Veridata reduces the amount of time verification takes and does high speed data www it ebooks info CHAPTER ARCHITECTURE verification while the databases are online Results are presented on a web interface Veri
128. 0 00 updated 02 24 26 ago Log Read Checkpoint Oracle Redo Logs 2011 03 13 23 45 30 Seqno 0 RBA 0 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE Target Extract Trails Remote Trail Name Seqno RBA Max MB AA 0 0 10 GGSCI oracledba 5 info exttrail aa Extract Trail AA Extract EXTORA Seqno 0 RBA 0 File Size 10M Oracle GoldenGate Parameter File Configuration Issues Oracle GoldenGate relies on parameter files to store and manage process operations for core functionality with the Extract for source data capture the Manager process for network communication and interprocess operations and the Replicat for target apply processing on the source system If these parameter files aren t set up correctly or are unavailable then Oracle GoldenGate processing will fail The first scenario involving parameter file configuration issues is to check whether the parameter file is located in the correct directory By default parameters are stored in the dirprm subdirectory after Oracle GoldenGate has been installed You need to verify that the parameter files for the Extract and Replicat are in this directory and that these parameter files have the same name as the process group for Oracle GoldenGate In the event that the parameter file is missing or not in this directory perform a search to locate the correct parameter file If you don t remember where you placed the configuration parameter file for the process group y
129. 00 13 0 02 0 00 25 0 28 0 00 13 Majflt Minflt Sort Procs Aio Procs QueueSize RegionKB 0 0 0 2 0 0 512 File Name 1 0 Mode Busy Wait MB sec Xfers Bytes Records Input Reads release records buffered 100 1 08 1 08 9 2186324 73236 Output Writes return records buffered 100 0 00 218 63 9 2186324 73236 Using Vericom Command Line Vericom command line scripts are usually used to automate the scheduled jobs It can be used to override the settings in groups compare pairs and profiles For example you can override the sql predicates with system dates to compare the most recent data only Set up number of threads depending on the day or night jobs By default Vericom will run the job in the background and it will not wait for the job to finish before returning the prompt C Weridata server bin gt vericom j job big table Connecting to localhost 4150 Run ID 1035 0 0 The exit code will be zero if the job started successfully but it will not tell you if any compares fail 185 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA If you want to kick off the job to capture the detail messages and status you can use either w or wp arguments wp lt number gt is poll interval in minutes C Weridata server bin gt vericom j job big table w Connecting to localhost 4150 Run ID 1035 0 0 Job Start Time 2011 05 01 22 12 57 Job Stop Time 2011 05 01 22 13 02 Job Report Filename C 00001035 job big table rpt Num
130. 01 20 14 04 dirdat 12000003 2011 01 20 13 57 2011 01 20 13 57 dirdat 12000003 2011 01 20 13 57 2011 01 20 13 57 dirdat 12000003 2011 01 12 10 59 2011 01 20 13 57 dirdat 12000001 2011 01 07 08 38 2011 01 12 10 59 www it ebooks info CHAPTER 4 M BASIC REPLICATION dirdat 12000000 Initialized 2011 01 07 08 38 dirdat 12000000 Initialized First Record dirdat 12000000 Initialized First Record Current directory gger ggs Report file gger ggs dirrpt RHREMD1 rpt Parameter file gger ggs dirprm RHREMD1 prm Checkpoint file gger ggs dirchk RHREMD1 cpr Process file gger ggs dirpcs RHREMD1 pcr Stdout file gger ggs dirout RHREMD1 out Error log gger ggs ggserr log Finally you can run the STATS command on your Replicat This shows if the Replicat has actually processed any DML activity In the example the Replicat has processed four inserts and two deletes since it was started You may also notice the Replicat has handled two insert collisions because you had to turn on the HANDLECOLLISIONS parameter to resolve errors Let s do a STATS command in the following example on the Replicat RHREMDI to see if any data has been processed GGSCI targetserver 2 stats rep RHREMD1 Sending STATS request to REPLICAT RHREMD1 Start of Statistics at 2011 01 18 18 50 38 Replicating from HR EMPLOYEES to HR EMPLOYEES Total statistics since 2011 01 15 13 54 52 Total inserts 4 00 Total updates 0 00 Total deletes 2 00 Total discard
131. 09 Transparent Data Encryption TDE 281 trigger based 2 triggers disabling 57 58 troubleshooting 217 240 common problems and solutions 217 218 configuration issues 226 234 database availability 227 228 incorrect software versions 227 missing process groups 228 missing trail files 228 with network 231 232 with operating system 230 231 with parameter files 229 230 database issues character set configuration issues 239 data pump errors 236 data synchronization issues 238 239 Extract process 235 fetch failures 239 240 INDEX missing column errors 239 Replicat process 236 238 tables missing key constraints 239 discard file 223 225 missing 224 not opening 225 too big 224 error log analysis 223 Logdump utility 296 300 accessing 296 getting help with syntax 296 299 HISTORY command 299 opening trail files with 299 300 process failures 218 221 Extract process 219 221 without report diagnostics 221 trace commands 225 226 process tracing with TLTRACE parameter 225 TRACE parameters 226 troubleshooting case study 226 trail file problems 221 223 not emptying 221 222 not rolling over 222 purging issues 222 Truncate Trailing Spaces When Comparing Values option 181 tuning 127 152 329 www it ebooks info INDEX databases 289 290 designing and implementing solutions 134 152 BATCHSQL parameter 146 149 database 152 disk storage 149
132. 1 ORA 04088 error during execution of trigger HR REPORA INSERT Operation failed at seqno 15 rba 28483311 Problem replicating HR EMP to HR TRGTREP Error occurred with insert record target format A_TIMESTAMP 2011 03 09 11 28 11 OK 1 0000 NOTOK 1 0000 Discard records provide useful clues for issues that occur during processing so you can identify and solve the root cause quickly Discard File Not Created Sometimes you may encounter the frustrating issue that Oracle GoldenGate fails to create a discard file One common cause is that the discard file s location wasn t provided in the parameter file for use with the DISCARDFILE parameter GoldenGate doesn t create a discard file by default so you need to give a file name and location of the discard file otherwise it won t be created second cause of missing discard files occurs when the DISCARDFILE parameter doesn t reference the correct directory on the file system You need to verify that sufficient read and write access permissions are granted for the discard file and the directory that contains the discard file In addition make sure the user who accesses the discard file has the required security permissions to access read and write to and from the file location Discard File Is Too Big Keep in mind that Oracle GoldenGate constantly writes to the discard file as processing takes place If the discard file isn t given sufficient disk space on the file sy
133. 1 Sending GETLAG request to EXTRACT PPREMP1 Last record lag 6 seconds At EOF no more records to process GGSCI oldserver STOP EXTRACT PPREMP1 4 After verifying that there are no more records to process stop the Replicat as shown here GGSCI newserver LAG REPLICAT RPREMP1 Sending GETLAG request to REPLICAT RPREMP1 Last record lag 7 seconds At EOF no more records to process GGSCI newserver STOP REPLICAT RPREMP1 Now the old database is no longer being used by the application and the replication from the old database to the new database is stopped You can proceed to part 2 see Figure 13 5 and start the fallback Extract and switch the application to use the new database Start the Replicat Start the Pump Start the Extract T 6 5 C 9 Na ME Old ra Local Qa New Database Replicat lt 1 Data Pump it Extract Database Wa penc Remote Source D Trail Trail atabase Application Start the Application 1 q Figure 13 5 Zero downtime migration cutover part 2 275 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION 276 Following are descriptions for steps 1 4 as shown in Figure 13 5 1 To begin capturing changes made to the new database in preparation for the application cutover alter the Local Extract for the new database to begin capturing changes and then start it as follows GGSCI newserver gt ALTER EXTRACT LPREMP2 BEGIN NOW GGSCI newserver
134. 1 0 Windows Oracle GoldenGate Oracle Goldengate Director XP Director 11g Server and Client Windows Oracle 11g GoldenGate Veridata XP Veridata We will also use examples from the Linux platform in this book so you can follow along with either Windows Linux or both setups if you desire Table 2 2 shows the test Linux configuration that will be used Table 2 2 Oracle Linux Test Environment for Oracle 11g GoldenGate Environment 0 Database Software Function OL11SRC Oracle Oracle Oracle 11g GoldenGate Source System Linux 5 11 2 v11 1 1 0 OL11TRG Oracle Oracle Oracle 11gGoldenGate Target System Linux 5 11 2 v11 1 1 0 Director Windows Oracle GoldenGate Oracle Goldengate Director XP Director 11g Server and Client Veridata Wi ndows Oracle 11g GoldenGate Veridata XP Veridata Table 2 3 shows the configuration that we ll use in our upcoming chapter on heterogeneous replication The term hetergenous refers in this case to replication across database brands www it ebooks info 13 CHAPTER 2 INSTALLATION 14 Table 2 3 Windows Test Environment for Oracle 118 GoldenGate for Homogeneous Replication Environment OS Database Software Function MYSQLS Windows MySQL 5 x Oracle 11g GoldenGate Source System XP v11 1 1 0 MySQL WIN11TRG Windows Oracle 11gR2 Oracle 11gGoldenGate Target System XP v11 1 1 0 Oracle WIN11SRC Windows Oracle 11gR2 Oracle GoldenGate Source System XP v11 1 1 0 Oracle MSSQL2K8 Wi
135. 11 01 12 10 59 dirdat 11000004 2011 01 07 08 38 2011 01 07 08 38 dirdat 11000004 Initialized 2011 01 07 08 38 dirdat 11000000 Initialized First Record Current directory gger ggs ora10 Report file gger ggs dirrpt PHREMD1 rpt Parameter file gger ggs dirprm PHREMD1 prm Checkpoint file gger ggs dirchk PHREMD1 cpe Process file gger ggs dirpcs PHREMD1 pce Stdout file gger ggs dirout PHREMD1 out Error log gger ggs ggserr log Finally you can run the STATS command on your data pump Extract This shows whether the data pump Extract has actually processed any DML activity In the example the data pump Extract has processed four inserts and two deletes since it was started GGSCI sourceserver 2 stats ext PHREMD1 Sending STATS request to EXTRACT PHREMD1 Start of Statistics at 2011 01 18 18 50 38 Output to gger ggs dirdat l2 Extracting from HR EMPLOYEES to HR EMPLOYEES 69 www it ebooks info CHAPTER 4 M BASIC REPLICATION 70 Total statistics since 2011 01 15 13 54 52 Total inserts 4 00 Total updates 0 00 Total deletes 2 00 Total discards 0 00 Total operations 6 00 After you ve started the Local Extract and the data pump Extract you can run the initial data load from your source to the target databases You can use either GoldenGate itself or the DBMS vendor load utilities to do the initial data load The DBMS vendor load utilities are optimized to load the vendor database so you may find that they
136. 111 e18482 pdf To perform the Oracle GoldenGate Director Server software installation execute the setup script called ggdirector serverset version and follow the screen prompts Install Oracle GoldenGate Director Now it is time to install the Oracle GoldenGate Director software The following steps must be performed to ensure a successful installation of the Oracle GoldenGate Director software 1 Download the Oracle 11g GoldenGate Director software 2 Download and install the Microsoft Visual C 2005 SP1 Redistributable Package from http microsoft com 3 Install and configure Oracle 11g Weblogic Standard Server on the Oracle GoldenGate Director Server 4 Download Oracle 11g database software to the Oracle 11g GoldenGate Director Server This will be used for the repository database with the Director Server Installing Oracle GoldenGate Veridata Oracle GoldenGate Veridata performs data synchronization functionality that allows you to ensure that all of the source and target Goldengate environments are in sync in terms of data replicated between source and target Veridata uses the concept of a compare pare set of algorithms to measure synchronization changes between source and target Goldengate environments Veridata consists of one or more agents and the repository server that houses the metadata objects for Oracle GoldenGate Veridata operations For a standard Oracle GoldenGate Veridata installation you will need to instal
137. 150 GROUPTRANSOPS parameter 149 network 150 151 parallel Extract and Replicat processes 134 135 RMTHOST parameter 151 determining problems 132 133 methodology 127 128 performance creating baseline 129 131 defining requirements 128 129 evaluating current 131 132 profiles settings 180 two phase confirmation 174 U universal database See IBM DB2 UDB UNIX operating system installation of Oracle GoldenGate IBM DB2 UDB 23 24 Sybase DBMS 23 Teradata RDBMS platform 21 22 330 unplanned failover disaster recovery replication for 256 use cases 40 45 USECHECKPOINTS option 90 223 User Defined option Column Mapping screen 180 USERID parameter configuring for initial load Replicat file 73 for Local Extract 61 62 for Replicat process 79 users dedicated creating 281 specifying 102 utilities tools and 45 47 DEFGEN 46 Director 46 47 GGSCI 46 Logdump 46 Reverse 46 Veridata 46 UTL_FILE 103 v command GGSCI 227 V V TRANSACTION dynamic performance view 235 var log messages 162 var log messages directory 213 www it ebooks info vericom command line jobs 178 Vericom interface 173 185 186 vericon exe job job_big_table command 179 Veridata tool 46 171 188 benefits of 174 comparison 173 179 Compare Pairs 177 178 database connections 175 176 of different column types and compare formats 182 groups 177 jobs 178 179 profiles 179 of re
138. 1G2 ORADATA WIN11SRC REDO01 LOG for sequence 16 Not able to establish initial position for begin time 2011 03 16 00 50 47 2011 03 16 00 51 55 ERROR OGG 01668 PROCESS ABENDING You can find out the log file that the Extract is looking for by issuing the GGSCI VIEW REPORT lt group gt command Your best bet to resolve missing archive logs with the Extract is to restore the missing archive log files and then restart the Extract group To avoid these errors you should maintain sufficient archive logs available to the Extract until the data contained in these logs has been processed second issue with Extracts and Oracle database archive log files occurs whenever the archive logs aren t stored in the default Oracle location In this case you can use the Extract parameter ALTARCHIVELOGDEST path name to use another location for the archive logs that the Extract needs for processing One key way you can identify a missing archive log issue is that when this occurs the Extract startup process runs slowly and appears to be in a hang condition as it looks back for the archive log sequences to find the specific logs that it requires The Extract must search through all past logs if it was restarted while a database transaction was opened thus creating the requirement to search and go back to past operations until a commit was received because only committed transactions are recorded by Oracle GoldenGate In flight transactions aren t captured and also are
139. 2 4 251 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION Stop the Extract Stop the Pump Stop the Replicat 2 3 4 iwi oe e Primary Local vt D Standby Database ri T Replicat Database Remote Database Trail Trail Application Stop the Application 1 Figure 12 4 Disaster recovery planned switchover part 1 Following are descriptions for the steps in Part 1 as shown in Figure 12 4 1 Stop the application connected to the primary database 2 After verifying there is no lag stop the Local Extract as shown in the following example GGSCI primaryserver gt LAG EXTRACT LHREMD2 Sending GETLAG request to EXTRACT LHREMD2 Last record lag 10 seconds At EOF no more records to process GGSCI primaryserver gt STOP EXTRACT LHREMD2 3 After verifying there is no lag stop the data pump Extract as shown in the following example GGSCI primaryserver gt LAG EXTRACT PHREMD2 Sending GETLAG request to EXTRACT PHREMD2 Last record lag 6 seconds At EOF no more records to process GGSCI primaryserver gt STOP EXTRACT PHREMD2 4 After verifying there are no more records to process stop the Replicat as shown in the following example GGSCI standbyserver gt LAG REPLICAT RHREMD2 Sending GETLAG request to REPLICAT RHREMD2 Last record lag 7 seconds At EOF no more records to process GGSCI standbyserver gt STOP REPLICAT RHREMD2
140. 22 23 24 Figure 8 3 Average lag in seconds by hour 167 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE Checking Memory and CPU Scripts If memory and CPU are the weakest links in your system then this script will be handy to report memory and CPU usage every hour It e mails you the memory and CPU usage for all the GoldenGate Extract processes as well as the total free memory Note that goldengate is the Unix Linux user id Here s the script bin ksh EMAIL_LIST shingchung aol com 5555555555 vtext com host hostname cd ggs scripts rm top log rm top rpt top h f top log n 50 cat top log grep extract gt top rpt echo gt gt top rpt echo Free Memory gt gt top rpt vmstat 1 2 tail 1 awk printf d s n 5 4 1024 MB gt gt top rpt mailx s GoldenGate Memory Usage on host EMAIL_LIST lt ggs scripts top rpt Output 19 14000 goldengate 152 20 2331M 1201M run 602 09 31 64 31 59 extract 10 13655 goldengate 137 20 80912K 16432K sleep 1568 25 5 97 5 96 extract Free Memory 47610MB The Extract process is using 31 59 of one CPU and the data pump is using 5 96 The Extract usually consumes more CPU than the data pump Memory usage for the Extract is around 2331MB total and 1201MB in resident memory The entire server memory is 47GB so you still have plenty of memory free The GoldenGate Extract process usually consumes 1 to 5 of overall CPU utilization
141. 271 Go to Compare Pair Configuration link 178 GoldenGate Director Admin Tool 192 GoldenGate Director defined 189 GoldenGate GGSCI Manager processes 189 GoldenGate Manager process 49 GoldenGate RANGE function 288 GoldenGate rate monitoring 158 159 GoldenGate Reference Guide 60 GoldenGate Replication Environment 51 GoldenGate Software Command Interface GGSCD 46 189 190 GRANT DBADM ON DATABASE TO USER lt ggs_user gt 24 graphical user interfaces GUIs modifying Manager parameter file using 194 195 Group Configuration screen 177 178 Group parameter 71 groups 177 228 GROUPTRANSOPS parameter 149 150 GUIs graphical user interfaces modifying Manager parameter file using 194 195 H HANDLECOLLISIONS parameter 53 77 79 80 83 247 273 HELP ADD EXTRACT 68 Help command 46 HELP command 293 294 heterogeneous replication 111 125 Microsoft SQL Server to Oracle 111 118 database configuration 113 118 initial data load completion 113 preparing environment 113 source database configuration 114 sample Microsoft SQL Server database 118 124 verifying operational readiness 124 125 HISTORY command 285 299 Host is Observable option in Director Admin Tool 193 host name 231 HP OpenView 234 IBM DB2 UDB universal database to Oracle replication 124 version 8 x 124 www it ebooks info version 9 x 124 on Windows and UNIX installation of Oracle G
142. 3 CHAPTER 7 88 TUNING 134 Designing and Implementing a Solution Now you re at the point where you can move forward to design and implement a solution to tune the replication At this point you may have several possible solutions to the problem You need to go through some analysis to determine which solution to implement first One solution may be easier and have less impact to implement so you may try it first to see if it works or rule it out The important thing is to implement only one tuning change at a time and then measure the impact That way you have a better indication that your solution actually solved the problem and improved performance If you implement multiple solutions at once it may be difficult to determine which one fixed the problem or made it worse The following sections go through some different tuning strategies for Extracts and Replicats Of course these aren t the only strategies and you may discover your own based on your particular environment and experiences Because the scope of this book is GoldenGate it only covers tuning of components such as the database and servers as they relate to GoldenGate replication but it s important to rule out general performance issues with these components before tuning GoldenGate Using Parallel Extracts and Replicats Typically most tuning is focused on the Replicats and not the Extracts The reason is that the Extracts are usually reading from the database transacti
143. 3 columns comparison of different types of 182 excluding 180 filtering 95 96 mapping 97 99 generating data definitions file 97 98 303 www it ebooks info INDEX transforming columns 98 99 missing 239 command line of Vericom interface 185 186 commands Logdump utility 296 300 accessing 296 getting help with syntax 296 299 HISTORY command 299 opening trail files with 299 300 quick guide to 292 300 ADD command 293 GGSCI command 293 HELP command 293 294 INFO command 294 295 SEND command 295 STATUS command 295 compare formats comparison of different 182 compare methods 180 Compare Pair Configuration screen 181 182 Compare Pair Mapping screen 178 Compare Pair Reports 184 Compare Pairs 177 178 comparison of different column types and compare formats 182 304 of incremental data for large tables 181 182 of real time replication data 182 using Veridata tool 173 179 Compare Pairs 177 178 database connections 175 176 groups 177 jobs 178 179 profiles 179 tables and data scripts 176 177 components 34 40 capture process 35 36 Collector process 38 data pump 36 38 databases source 34 35 target 39 40 delivery 39 Local Extract process 35 36 Manager process 40 network 38 Replicat process 39 trails remote 38 39 source 36 COMPRESS option 151 compressed row hash data 171 www it ebooks info compressupdates parameter 209
144. 3 20 13 19 23 000000 RBA 8144 5 When the Replicat RHREMDI has processed all the remaining changes from the current data pump Extract trail and has zero lag showing stop it and record the EXTRBA and EXTSEO from the info command You need those values when you add the new Replicats so they begin processing at the correct location in the data pump Extract trail file 12 From the info command you can see that EXTSEO is equal to the trail file number of 13 EXTRBA is 2351 GGSCI targetserver 1 info rep RHREMD1 REPLICAT RHREMD1 Last Started 2011 03 20 13 18 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoint File dirdat 12000013 2011 03 20 13 19 23 000107 RBA 2351 GGSCI targetserver 2 stop rep RHREMD1 Sending STOP request to REPLICAT RHREMD1 Request processed 6 Add the new Replicat RHREMD2 and tell GoldenGate to begin processing at the EXTRBA and EXTSEO in the Local Extract trail 12 that you recorded in step 5 Be aware that if you don t tell GoldenGate to start at a specific location you can end up re replicating changes from the first trail and cause collisions in the Replicat 144 www it ebooks info CHAPTER 7 88 TUNING GGSCI targetserver gt ADD REPLICAT RHREMD2 EXTTRAIL dirdat 12 EXTSEONO 13 EXTRBA 2351 GGSCI targetserver gt info rep RHREMD2 REPLICAT RHREMD2 Initialized 2011 03 20 22 51 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoin
145. 5 ADVANCED FEATURES Switching to next trail file dirdat 12000007 at 2011 01 28 23 20 02 due to EOF with current RBA 1281 Opened trail file dirdat 12000007 at 2011 01 28 23 20 02 Processed extract process graceful restart record at seq 7 rba 1191 Processed extract process graceful restart record at seq 7 rba 1253 In addition to the standard reports you can add parameters to instruct the Extract or Replicat to periodically add more information to the reports As shown in the example add the following parameters to your Extracts and Replicats to enhance the default reporting ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 The following sections look at each of these parameters in more detail REPORTCOUNT You can use the REPORTCOUNT parameter to have the Extract or Replicat automatically add a count of the records GoldenGate has processed since startup In the example you request a record count every 30 minutes Here is an example of what the reporting output looks like 1000 records processed as of 2010 01 28 11 30 40 rate 154 delta 215 Since the Extract process started it has processed 1 000 records as of the time displayed 11 30 40 You also asked for RATE information to be printed in the report The rate calculations are as follows Rate of records processed since startup total time since startup Delta of records since last report time since last report In the example the proc
146. 88 TUNING 140 10 Add the new Replicat RHREMD2 and tell GoldenGate to use the new 13 trail from data pump Extract PHREMD2 for processing Because 13 is a new trail you can take the default to start processing from the beginning of the first 13 trail Run an info command to verify the Replicat has been added properly GGSCI targetserver ADD REPLICAT RHREMD2 EXTTRAIL dirdat 13 GGSCI targetserver info rep REPLICAT RHREMD1 Last Started 2011 03 20 13 18 Status STOPPED Checkpoint Lag 00 00 00 updated 00 52 11 ago Log Read Checkpoint File dirdat 12000013 2011 03 20 13 19 23 000107 RBA 2351 REPLICAT RHREMD2 Initialized 2011 03 20 22 41 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 03 ago Log Read Checkpoint File gger ggs dirdat 13000000 First Record RBA 0 Start the data pump Extracts and begin sending changes to the target server GGSCI sourceserver start ext PHREMD Sending START request to MANAGER EXTRACT PHREMD1 starting Sending START request to MANAGER EXTRACT PHREMD2 starting GGSCI sourceserver info ext PHREMD EXTRACT PHREMD1 Last Started 2011 03 20 22 51 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 06 ago Log Read Checkpoint File dirdat 11000018 First Record RBA 8144 EXTRACT PHREMD2 Last Started 2011 03 20 22 51 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 06 ago Log Read Checkpoint File dirdat 11000018 First Record RBA 8144 Start both Replicats
147. 9 DetailReportView role name Tomcat Web Server Administration Tool screen 187 diagnostics report process failures without 221 307 www it ebooks info INDEX 308 dirdat directory 62 90 dirdef directory 98 Director 11g Oracle GoldenGate See Oracle GoldenGate Director 11 Director Admin Tool 193 Director Administrator component 190 Director application 189 214 advanced mapping 210 212 alerts 212 214 altering RUN options for Extract and Replicat objects 206 changing trail file size 206 components of 189 192 Director Administrator component 190 Director Client component 192 Director Server and database component 190 Director Web component 191 GGSCI instances 190 Data Pump process 204 206 data source 192 193 extracting tranlogoptions 206 207 finding parameters or attributes in 209 210 generating definition files 207 209 Initial Load task 195 200 one way replication 200 203 Director Client component 192 Director component 189 190 Director environment 13 Director Monitor Agent 190 Director repository database 190 Director Server and database component 190 Director tool 46 47 Director Web component 191 directories installation 282 diroby directory 63 81 285 dirprm subdirectory 59 77 91 229 dirrpt directory 133 disaster recovery replication 241 262 for planned switchover 251 256 prerequisites for 241 242 requirements for 242 243 sett
148. A team or a separate team If the GoldenGate user ID isn t owned by the DBA team then the GoldenGate responsibilities need to be divided between the DBA team and the GoldenGate team based on security required 281 www it ebooks info CHAPTER 14 TIPS AND TRICKS Encrypting Passwords You should encrypt the GoldenGate database user passwords in any secured environment The following example shows how to use GoldenGate to encrypt the password abc and specify it in the parameter file The example uses a default encryption key but you can use a specific key if needed GGSCI server 1 gt encrypt password abc No key specified using default key Encrypted password AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB After you determine the encrypted value for the key you can specify it in the Extract or Replicat parameter file as shown in the following example USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default Next let s review using a dedicated GoldenGate installation directory Creating a Dedicated Installation Directory You should install the GoldenGate software in its own separate directory which is owned by the dedicated GoldenGate operating system user This keeps the GoldenGate software and other files separate and allows them to be located and monitored quickly and easily Because the GoldenGate software is specific to a DMBS vendor release you may want to qualify the subdirectory by the DBMS release Example ins
149. ANCED FEATURES 104 SOL connect as sysdba Connected SOL marker_setup sql Marker setup script You will be prompted for the name of a schema for the GoldenGate database objects NOTE The schema must be created prior to running this script NOTE Stop all DDL replication before starting this installation Enter GoldenGate schema name gger Marker setup table script complete running verification script Please enter the name of a schema for the GoldenGate database objects Setting schema name to GGER MARKER TABLE Script complete Run the ddl 591 from the GoldenGate software installation directory This script is included as part of the GoldenGate software installation SOL Qddl setup sql GoldenGate DDL Replication setup script Verifying that current user has privileges to install DDL Replication Checking user sessions Check complete You will be prompted for the name of a schema for the GoldenGate database objects NOTE For an Oracle 10g source the system recycle bin must be disabled For Oracle 11g and later it can be enabled NOTE The schema must be created prior to running this script NOTE Stop all DDL replication before starting this installation Enter GoldenGate schema name gger You will be prompted for the mode of installation To install or reinstall DDL replication enter INITIALSETUP To upgrade DDL replication enter NORMAL Enter mode of installation INITIALSETUP www i
150. ARGETDEFS parameter is commented out and you include a new SOURCEDEFS parameter that points to the defgen file AssumeTargetDefs SourceDefs dirdef RHREMD1 defs Now that you ve set up the column mapping and generated a data definitions file you re almost ready to start the Replicat The next section addresses one additional requirement Transforming Columns You must do one more thing before you can start replicating the EMPLOYEES table to the Staff table The WAGES column in the target Staff table needs to reflect the annual employee salary The SALARY column in the source database EMPLOYEES table stores only the monthly salary So during replication you must transform the monthly salary amount into an annual salary amount You can do that using the Replicat MAP and COLMAP parameters as shown in the following example Map HR EMPLOYEES Target HR STAFF COLMAP USEDEFAULTS WAGES COMPUTE SALARY 12 The example maps the monthly SALARY column multiplied by 12 months into the annual WAGES column during replication You can use the built in GoldenGate COMPUTE function for the calculation Another difference between the source and target EMPLOYEES tables is that the target has an additional column for FULL NAME The full name is a concatenation of the LAST NAME a comma and the FIRST NAME You can perform this transformation using the Replicat MAP parameter as shown here Map HR EMPLOYEES Target HR STAFF COLMAP USEDEFAULTS WA
151. CT parameter must be the first entry in the parameter file and the EXTTRAIL parameter must precede any associated TABLE statements 61 www it ebooks info CHAPTER 4 M BASIC REPLICATION Next let s look at the EXTTRAIL parameter EXTTRAIL You can use the EXTTRAIL parameter to specify the two character local Extract trail file name As discussed in Chapter 3 trail files are staging files used to store the committed transactions In the example the trail file holds the HR transactions extracted from the transaction log by the Local Extract Next the trail is processed by the data pump and sent across the network to the target server Trail files are usually written to the dirdat directory underneath the GoldenGate software installation directory In this case you re using a trail file named 11 GoldenGate automatically appends six characters to the trail file name for aging purposes For example the first trail file is named 11000000 in the dirdat directory As this trail file fills up GoldenGate will create the next new trail file named 11000001 then 11000002 and so on You can specify a FORMAT RELEASE option with the EXTTRAIL parameter to instruct GoldenGate to write the trail file in an older release format for backward compatibility This may be useful for example if you have an older version of GoldenGate on another system that you need to process trail files using a more recent version of GoldenGate For SQL Server you sh
152. D AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD3 dsc Append DiscardRollover at 02 00 ON SUNDAY Map HR EMPLOYEES Target HR EMPLOYEES COLMAP USEDEFAULTS FILTER RANGE 3 3 Map HR JOBS Target HR JOBS COLMAP USEDEFAULTS FILTER 3 3 HR JOB HISTORY Target HR JOB HISTORY COLMAP USEDEFAULTS FILTER GRANGE 3 3 143 www it ebooks info CHAPTER 7 88 TUNING Map HR LOCATIONS Target HR LOCATIONS COLMAP USEDEFAULTS FILTER RANGE 3 3 Map HR DEPARTMENTS Target HR DEPARTMENTS COLMAP USEDEFAULTS FILTER GRANGE 3 3 Map HR COUNTRIES Target HR COUNTRIES COLMAP USEDEFAULTS FILTER GRANGE 3 3 Map HR REGIONS Target HR REGIONS COLMAP USEDEFAULTS FILTER 3 3 4 Stop the existing data pump Extract PHREMDI You want it to stop extracting so you can make sure the lag on the Replicat goes down to zero and you can continue with the Replicat changes The Local Extract can continue to run Also run an info command to verify that it s stopped GGSCI sourceserver 1 stop ext PHREMD1 Sending STOP request to EXTRACT PHREMD1 Request processed GGSCI sourceserver 2 info ext PHREMD1 EXTRACT PHREMD1 Last Started 2011 03 20 13 09 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 10 ago Log Read Checkpoint File dirdat 11000018 2011 0
153. D POS INSTR STR1 I TO EMAIL SIR2 gt POS gt 1 NTH gt L EMAIL L LAST EMAIL ADDR L EMAIL END POS 0 L EMAIL END POS CASE WHEN L EMAIL END POS O THEN LENGTH I TO EMAIL 1 WHEN L EMAIL START POS O THEN L EMAIL END POS 1 ELSE L EMAIL END POS END L EMAIL SUBSTR STR1 TO EMAIL POS EMAIL START POS LEN gt EMAIL END POS L EMAIL START POS gt I gt L UTL_SMTP RCPT C L_EMAIL EXIT WHEN L LAST EMAIL ADDR EMAIL START POS L EMAIL END POS 1 END LOOP UTL SMTP OPEN DATA C SEND HEADER From I FROM NAME I SEND HEADER To Recipient I TO EMAIL SEND HEADER Subject I SUBJECT UTL SMTP WRITE DATA C UTL TCP CRLF I MESSEGE UTL SMTP CLOSE DATA C UTL SMTP QUIT C EXCEPTION WHEN UTL SMTP TRANSIENT ERROR OR UTL SMTP PERMANENT ERROR THEN BEGIN UTL SMTP QUIT C OM EMAIL 166 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE EXCEPTION WHEN UTL SMTP TRANSIENT ERROR OR UTL SMTP PERMANENT ERROR THEN NULL END RAISE APPLICATION ERROR 20000 Failed SOLERRM 3 END SEND_EMAIL Avg lag in Sec by day of week Avg Lag in Sec by day of week Figure 8 2 Average lag in seconds by day of week Avg Lag in Sec by hour Avg Lag in Sec by hour 123456 7 8 9 1011 12 13 14 15 16 17 18 19 20 21
154. DS 2 BEGIN NOW 101 www it ebooks info CHAPTER 5 ADVANCED FEATURES Tip If the number of threads ever changes such as adding new node and instance you must remember to drop and re add your Extracts with the correct number of threads Next let s review how to configure GoldenGate for Oracle ASM Configuring for Oracle ASM If you re using Oracle ASM some special GoldenGate parameters are needed in order to allow the GoldenGate Extract to connect to ASM In addition you must configure the Oracle listener and tnsnames ora to support the connection from GoldenGate Let s look at the GoldenGate parameters first Specifying the ASM User When using ASM the GoldenGate local Extract must log in to the ASM instance in order to access the redo logs You need to add an additional TranLogOptions ASMUser parameter in the local Extract parameter file to specify the ASM user ID and password as shown in the following example TranLogOptions ASMUser sys asm asmpassword encrypted password ENCRYPTKEY default The ASM user can be any user that has the SYSDBA privilege in the ASM instance or the SYSASM privilege in 11g onwards The example uses the sys user ID to connect to the ASM instance Updating the Listener After updating your GoldenGate local Extract parameter file you need to make sure your listener is listening for connections on the ASM instance Add a clause as shown in the following Oracle 11g example to your l
155. E GOLDENGATE ADMINISTRATOR 292 Oracle GoldenGate Sybase Installation and Setup Guide 11g Release 1 11 1 1 September 2010 Oracle GoldenGate DB2 LUW Installation and Setup Guide 11g Release 1 11 1 1 August 2010 Oracle GoldenGate Windows and UNIX Administrator s Guide 11g Release 1 11 1 1 August 2010 Oracle GoldenGate Windows and UNIX Reference Guide 11g Release 1 11 1 1 August 2010 Oracle GoldenGate Windows and UNIX Troubleshooting and Tuning Guide 11g Release 1 11 1 1 August 2010 Oracle GoldenGate Oracle Installation and Setup Guide 11g Release 1 11 1 1 Oracle GoldenGate Director Administrator s Guide 11g Release 1 11 1 1 Oracle GoldenGate Veridata Administrator s Guide version 3 0Teradata Replication Services Using Oracle GoldenGate Release 13 0 August 2010 Teradata www info teradata com In addition to the Oracle documentation it s recommend that you review the following support notes for Oracle GoldenGate available online with a valid Customer Support Identifier CSI account at http support oracle com Note 1063374 1 Why Do I See ERROR Failed To Open Data Source Note 1290427 1 GoldenGate 11 Doesn t Parse Note 966106 1 Considerations for Choosing to Use a Datapump Note 965703 1 My Checkpoint Table Is Corrupted The book also references the following articles available online via the OTN site at http otn oracle com Zero Downtime Database Upgrades Using Oracle
156. ER ARCHITECTURE 46 Tools and Utilities Now that you have an understanding of the GoldenGate flow and processes let s take a quick look at some of the GoldenGate tools and utilities The next sections provide a brief overview of the tools and utilities You learn more about these in later chapters GGSCI is the GoldenGate Software Command Interface GGSCI is delivered as part of the GoldenGate software and can be started from the software installation directory using the ggsci command From the GGSCI command prompt you can enter commands to manage all parts of the GoldenGate environment including Extracts Replicats trails the Manager and so on The Help command is built into GGSCI to give information about each command and its syntax DEFGEN You can use the GoldenGate DEFGEN utility to generate GoldenGate table definition files when your source and target tables are different if the source and target tables are identical it isn t necessary to run DEFGEN You can also use DEFGEN to generate a data definitions file for a data pump Extract that is doing data transformations on a middle tier server DEFGEN is included as part of the regular GoldenGate installation and is available in the GoldenGate software installation directory Logdump You can use the GoldenGate Logdump utility to view records in the GoldenGate trail files The Logdump utility is part of the regular GoldenGate software installation Logdump lets
157. File Recovery Checkpoint position of oldest unprocessed transaction in the data so urce Sequence 0 RBA 0 Timestamp 2011 03 16 00 50 47 000000 Redo File Current Checkpoint position of last record read in the data source Sequence 0 RBA 0 Timestamp 2011 03 16 00 50 47 000000 Redo File www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE Write Checkpoint 1 GGS Log Trail Current Checkpoint current write position Sequence 0 0 Timestamp 2011 03 16 00 53 45 022000 Extract Trail AA Header Version 2 Record Source U Type 4 Input Checkpoints 1 Output Checkpoints 1 File Information Block Size 2048 Max Blocks 100 Record Length 2048 Current Offset 0 Configuration Data Source 3 Transaction Integrity 1 Task Type 0 Status Start Time 2011 03 16 00 50 47 Last Update Time 2011 03 16 00 50 47 Stop Status G Last Result 0 The statistic to look for is Write Checkpoint an example of which is shown here Write Checkpoint 1 GGS Log Trail Current Checkpoint current write position Sequence 0 RBA 0 Timestamp 2011 03 16 00 53 45 022000 Extract Trail AA If the number of write checkpoints isn t increasing in Oracle GoldenGate it means the Extract process is unable to send data across the network to the trail file If you re using a data pump Extract on the source system then issuing the GGSCI command INFO EXTRACT for both Extract proc
158. GENEOUS REPLICATION 116 GGSMGR Properties Local Computer General Log Recovery Dependencies Service name Display name GGSMGR Description Path to executable C ggs mssqlserver ymgr exe j Startup type Automatic M Help me configure service startup options Service status Stopped Sm Stor Pause Resume You can specify the start parameters that apply when you start the service from here Start parameters o Figure 6 3 Checking status for manager process in Windows event viewer Since the manager process on the source system for Microsoft SQL Server has not been started we start the process You can also configure the manager process to run as a Windows service so that it starts automatically upon server bootup or you can set it to require manual intervention to start the manager Log in to the GGSCI interface and create the subdirectories for the Oracle GoldenGate software on the source system C ggs mssqlserver gt ggsci Oracle GoldenGate Command Interpreter for ODBC Version 11 1 1 0 0 Build 078 Windows optimized Microsoft SQL Server on Jul 28 2010 18 55 52 Copyright C 1995 2010 Oracle and or its affiliates All rights reserve GGSCI oracledba 1 gt create subdirs Creating subdirectories under current directory C ggs mssqlserver Parameter files C ggs mssqlserver dirprm created Report files C ggs mssqlserver dirrpt created
159. GER REPLICAT RHREMD3 starting GGSCI targetserver gt info rep RHREMD REPLICAT RHREMD1 Checkpoint Lag Log Read Checkpoint REPLICAT RHREMD2 Checkpoint Lag Log Read Checkpoint REPLICAT RHREMD3 Checkpoint Lag Log Read Checkpoint Last Started 2011 03 20 22 00 00 00 updated 00 00 04 File dirdat 12000013 2011 03 20 22 51 22 493945 Last Started 2011 03 20 22 00 00 00 updated 00 00 04 File dirdat 12000013 First Record RBA 2351 Last Started 2011 03 20 22 00 00 00 updated 00 00 04 File dirdat 12000013 First Record RBA 2351 53 Status RUNNING ago RBA 2351 53 Status RUNNING ago 53 Status RUNNING ago 10 After some changes are applied you can check the status of the Replicats to make sure they re processing properly You can check the RBA and make sure it s increasing You could also do a stats replicat tablename command and a stats extract tablename command to make sure the Extracts and Replicats are processing changes GGSCI targetserver info rep REPLICAT RHREMD1 Checkpoint Lag Log Read Checkpoint REPLICAT RHREMD2 Checkpoint Lag Log Read Checkpoint REPLICAT RHREMD3 Checkpoint Lag Log Read Checkpoint Using BATCHSQL Last Started 2011 03 25 23 00 00 00 updated 00 00 03 File dirdat 12000013 2011 03 25 23 21 48 001492 Last Started 2011 03 25 23 00 00 00 updated 00 00 03 File dirdat 12000013 2011 03 25 23 21 48 001492 Last Started 2011 03 25 23 00 00 00 updat
160. GES COMPUTE SALARY 12 FULL NAME QSTRCAT LAST NAME FIRST NAME Now let s look at the Extract and Replicat parameter files with the mapping and filtering changes shown in italics Here s the enhanced Extract parameter file www it ebooks info 5 GGSCI sourceserver 1 gt edit params LHREMD1 Extract LHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt LHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY ENCRYPTTRAIL ExtTrail dirdat 11 Table HR EMPLOYEES FILTER ON DELETE SALARY 12 1000 WHERE EMPLOYEE ID 100 Table HR JOBS Next is the enhanced Replicat parameter file GGSCI targetserver 1 edit params RHREMD1 Replicat RHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs SourceDefs dirdef RHREMD1 defs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY DECRYPTTRAIL Map HR EMPLOYEES Target HR STAFF COLMAP USEDEFAULTS WAGES COMPUTE SALARY 12 FULL NAME STRCAT LAST_NAME FIRST_NAME Map HR JOBS Target HR JOBS Oracle Specific DBMS Configuration Options ADVANCED FEATURES Let
161. GIN NOW command 63 67 bidirectional replication 42 43 106 110 excluding transactions for 108 109 www it ebooks info handling conflict resolution for 109 110 broadcast replication 44 built in editor modifying Manager parameter file using 193 business objectives 279 280 by table 158 capture process 35 36 cascade delete constraints disabling 57 58 C Code agent 173 CDC Change Data Capture 56 Central Processing Unit CPU scripts monitoring 168 certification matrix Oracle Support 112 c ggate sql2008 directory 282 C ggs src directory 19 C ggs trgt directory 19 Change Capture process configuring for replication 121 124 IBM DB2 UDB to Oracle 124 MySQL to Oracle 122 Sybase to Oracle 123 Teradata to Oracle 123 Change Data Capture CDC 56 change date column 182 character fields RTRIM on 181 INDEX character sets configuration issues with 239 verifying 283 Check Connection option in Director Admin Tool 193 checkemployee empduplicate 0 110 checkemployee empduplicate 0 110 CHECKPARAMS parameter 230 checkpoint tables 90 92 236 237 271 282 checkpoints 39 chmod command 229 chown command 229 Client component Director 192 client installation requirements for Oracle GoldenGate Director 11g 26 Clustered environments 18 Collector process 38 COLS list 95 COLS parameter 95 COLSEXCEPT parameter 96 Column Mapping screen 17
162. GS1 started CJ G iNFO localnost gg EXTGGS1 CAPTURE 4 1 11 10 03 20 PM 06901518 Oracle GoidenGate Capture for Oracie EXTGGS1 prm Positioned to Sequence 137 RBA 16522768 Apr 1 2011 8 38 08 PM J G iNFO localhost og EXTGGS1 CAPTURE 4 1 14 10 03 19 PM 06601515 Oracle GoldenGate Capture for Oracle EXTGGS1 prm Positioning to begin time Apr 1 2011 8 38 08 PM J nro localnost_gg EXTGGS1 CAPTURE 4 11 10 03 16 068601635 Oracle GoidenGate Capture for Oracle EXTGGS1 prm BOUNDED RECOVERY reset to initial or altered checkpoint Figure 10 2 Oracle GoldenGate Director Web on iPad 191 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR 192 Director Client On the desktop Director Client can create change delete the parameter and trail files It also displays online help for the available Oracle GoldenGate parameters Before using the Director desktop client or web application you have to set up the Manager process in a GGSCI instance and start it you can t do anything without Manager started and running Both Director Web and Director Client depend on the Manager process to control the Extract and Replicat Therefore you can t modify the Manager parameter files using either Director Client or Web You use the Director Client in this book Anything you can do in Director Web you also can do in Director Client Their interfaces are similar This chapter shows you how to set up some basic and advanced Oracle Golde
163. Gate hyperlink and you can download all of the required software packages for Oracle Goldengate including Director and Veridata Figure 2 9 Oracle GoldenGate Downloads Coherence Developer Tools This page contains links to the most popular versions of Oracle GoldenGate for the Oracle Database Event Diven Architecture Suite on various platforms The certification matrix contains the complete list of supported platforms GlassFish Server The complete set of Oracle GoldenGate software including versions for other databases and platforms is available at Oracle E Delivery com When prompted for the product pack select Oracle Fusion Middleware identity Management JRockit SOA Note If you are not able to find a specific database and platform combination contact Oracle dies Support Your service request must include the following details Operating System including version and chipset x86 x86 64bit 1464 etc Tuxedo Database including version and bit information 32bit or 64bit WebCenter Suite You must acceptthe OTN License Agreement to download this software from this page Weblogic Server Accept License Agreement Decline License Agreement Complex Event Processing Business Intelligence Foundation Oracle GoldenGate Data Integration Oracle GoldenGate v11 1 1 0 0 for Oracle 10g 64 biton Solaris 9 10 25 MB Oracle GoldenGate v11 1 1 0 0 for Oracle 10g on Linux x86 64 61 MB Oracle Gol
164. Gate supports a wide variety of replication topologies such as one way replication bidirectional replication peer to peer replication broadcast replication and integration replication Each of these topologies can be utilized to support many use cases The topologies and examples of some use cases are discussed in the following sections One Way Replication One way replication as shown in Figure 3 4 is the simplest topology and is often used for reporting or query offloading purposes Data is replicated from a single source to a single target in only one direction Changes to database data are only made at the source database The target database is read only 40 www it ebooks info CHAPTER ARCHITECTURE Source Target Database Database Figure 3 4 One way replication A one way configuration is useful for offloading intensive queries and reports from the source database It can also be used for maintaining a hot standby database for failover purposes Another concept you should keep in mind is that the source and target databases can be different database technologies For example the source database can be Oracle for OLTP processing and the target database can contain a filtered subset of tables from the source database using the SQL Server database for reporting and analysis purposes A typical GoldenGate configuration to implement one way replication is shown in Figure 3 5 Replicat gt Target 3 Database Rem
165. GoldenGate Environments eene 113 Initial Data Load Completion for Oracle GoldenGate Environments ses 113 Source Oracle Database 113 Target Microsoft SQL Server Database Configuration sene 115 Creating Sample Microsoft SQL Server Database esses 118 Configure Change Data Capture on 0 121 Verifying Operational Readiness 1 eeesessessre eren eren ennt 124 SUMMALY qt 125 Chapter 7 127 T nina x I n us 127 Defining the Performance Requirements cesses 128 Creating a Performance Baseline rana a 129 Evaluating the Current Performance eese 131 Determining the Problem M 132 Designing and Implementing a Solution eser 134 Using Parallel Extracts and Replicats
166. HE DOCUMENTATION LIBRARY SEARCH OTN ORACLE GOLDENGATE DOCUMENTATION ORACLE TECHNOLOGY NETWORK secure search x mi Release Notes Documentation G0 CONTACT THE DOCUMENTATIONTEAM Uparade Guide Administrators Guide Reference Guide Troubleshooting and Tuning Guide c tree Installation and Setup Guide DB2 LUW Installation and Setup Guide DB2 2 05 Installation and Setup Guide 082 for i Installation and Setup Guide MySQL Installation and Setup Guide Oracle Installation and Setup Guide SQL MX Installation and Setup Guide SQL Server Installation and Setup Guide Sybase Installation and Setup Guide Teradata Installation and Setup Guide TimesTen Installation and Setup Guide Licensing Information Figure 2 10 Oracle GoldenGate documentation repository Based on your specific platform and infrastructure design you will need to consult the appropriate guides shown above As a first stop we advise you to review all release notes to avoid potential issues in terms of errors before during and after the installation is completed With each release for Oracle Goldengate software new bugs are discovered and patches provided by the development team in time to address known issues Note Always consult the Release Notes before you install Oracle Goldengate software 15 www it ebooks info CHAPTER 2 INSTALLATION Installing Goldengate Before you install the Oracle Goldengate software for Windows an
167. I oracledba 3 gt alter exttrail aa megabytes 20 extract extora EXTTRAIL altered GGSCI oracledba 4 gt info exttrail Extract Trail AA Extract EXTORA Seqno 0 RBA 0 File Size 20M Trail File Purging Issues Trail files sometimes aren t purged correctly thus causing errors during processing of data between the source and target database environments Probably the most common source of failed purging operations with trail files occurs due to failure to use PURGEOLDEXTRACTS to manage the trail files in use by Oracle GoldenGate You should add this parameter to the Manager parameter file to prevent old trail files from accumulating too fast and thus causing problems with your Oracle GoldenGate configuration Keep in mind that you must grant the Oracle GoldenGate Manager user account the correct level of access permissions to read from and write to the trail files and file systems when using this Another cause of trail file purging issues is an obsolete Replicat group that may still be referencing the old trail file Oracle GoldenGate trail files aren t purged if other processes still read them If an old Replicat group is referencing a trail file you use the DELETE REPLICAT command from GGSCI to delete the obsolete Replicat process group so that the checkpoint records are deleted thereby allowing the Manager process to purge the trail files Keep in mind that if a checkpoint table is still used for the www it ebooks info CHAPTER 11
168. LDENGATE VERIDATA 188 Summary Oracle GoldenGate Veridata is a comparison tool that enables the comparison to be performed on changing data something that is impossible with any other tool It has very good security built in so it can be used by both IT and business users to protect business sensitive data Veridata can be used in both the command line and Web based client It is flexible enough to integrate with the existing Extract Transform and Load ETL and monitoring processes The performance of Veridata is impressive it has the right architecture to compare data between two databases with compressed row hash values With proper setup there is no other data comparison tool that can perform better than Veridata www it ebooks info 10 GoldenGate Director Oracle GoldenGate Director is a multi tiered GUI application that enables the development and monitoring of almost all Oracle GoldenGate processes With Director you don t need to remember the parameter and argument syntax In parameter files the order of the codes is important Director takes care of the order for you However you always can use Param File Editor to overwrite parameters manually It also offers context sensitive help that gives you high level assistance on what the parameters do In order to administer Oracle GoldenGate you have to open many SSH windows for each server on which GoldenGate is running With Director you no longer nee
169. LLISIONS as soon as possible after GoldenGate has completed processing past the point of the initial data load or past the problem situation You can do this by removing the HANDLECOLLISIONS parameter in the Replicat parameter file or adding the NOHANDLECOLLISIONS parameter and restarting the Replicat Another way to turn off HANDLECOLLISIONS is to use the SEND command as shown next to communicate directly with the running Replicat If you choose this method don t forget to also go back and remove the HANDLECOLLISIONS parameter from your Replicat parameter file so you don t inadvertently turn it on again the next time the Replicat is restarted The following example sets the RHREMDI Replicat to use NOHANDLECOLLISIONS 79 www it ebooks info CHAPTER 4 M BASIC REPLICATION 80 GGSCI targetserver gt send replicat RHREMD1 NOHANDLECOLLISIONS HR Sending NOHANDLECOLLISIONS request to REPLICAT RHREMD1 RHREMD1 NOHANDLECOLLISIONS set for 1 tables and 1 wildcard entries The reason for turning off HANDLECOLLISIONS is that under normal processing conditions you want GoldenGate to create an error condition if a collision occurs and to stop processing There may be a serious problem that needs to be investigated In those cases you need to research the root cause of the problem rather than have HANDLECOLLISIONS mask the problem and GoldenGate continue processing You can see if the Replicat encountered any collisions by using the STATS REPLICAT c
170. M BASIC REPLICATION running and are waiting to be applied by the Replicat After you ve configured added and successfully started the Replicat the changes from the trail are applied to the target tables In some cases you may run into errors with duplicate or missing data caused by the ongoing changes made during the initial load process For example if a row was updated and then deleted during the initial load GoldenGate may try to apply the update to the missing row that was never loaded You can make GoldenGate try to resolve these errors automatically by adding the HANDLECOLLISIONS parameter to the Replicat You may also notice that it takes some time for the Replicat to catch up with applying all the changes made during the load particularly for a large database with heavy change volume and a long load time When the initial changes have been applied and there is no GoldenGate lag left the databases are fully synchronized At this point the Local and data pump Extracts and Replicat can continue to run and keep the databases synchronized in real time with the ongoing changes This section covers the various ongoing change Replicat configuration settings and options After that you see how to start the Replicat and verify that it s properly applying changes to the target Let s begin with configuring the Replicat Configuring the Replicat Before starting to configure the Replicat you should go back and double check the prerequisit
171. NG for the Local Extract and the Replicat If NLS LANG is incorrect it can cause the replication to fail or replicate incorrect data For the Replicat NLS LANG needs to match the NLS character set of the data in the trail file it s reading which was created from the source database You can set other environment variables in addition to NLS LANG using the SETENV parameter For example you can set ORACLE HOME and ORACLE SID as shown next In the example I chose instead to set the Oracle environment using the Oracle network TNS alias of TargetDB instead of using SETENV You can also set ORACLE HOME and ORACLE SID prior to entering GGSCI and not use a TNS alias or SETENV Any of these methods will work as long as the variable values are valid SETENV ORACLE HOME u01 app oracle product 11 1 0 SETENV ORACLE SID TargetDB Next let s review the USERID parameter www it ebooks info CHAPTER 4 M BASIC REPLICATION USERID Using the USERID parameter you can specify the GoldenGate database user ID and password to connect to the target database The GoldenGate password can be encrypted Chapter 5 discusses encryption For SQL Server you can use the SOURCEDB parameter with the USERID option as shown here SOURCEDB sqlserver userid sa PASSWORD userpw SOURCEDB specifies the SQL Server ODBC data source and USERID specifies the database user ID If you re using Windows authentication you can leave off the USERID and PASSWORD options
172. Nanda Editorial Board Steve Anglin Mark Beckner Ewan Buckingham Gary Cornell Jonathan Gennick Jonathan Hassell Michelle Lowman Matthew Moodie Jeff Olson Jeffrey Pepper Frank Pohlmann Douglas Pundick Ben Renow Clarke Dominic Shakeshaft Matt Wade Tom Welsh Coordinating Editor Anita Castro Copy Editor Tiffany Taylor Elizabeth Berry and Mary Behr Production Support Patrick Cunningham Indexer BIM Indexing and Proofreading Services Artist April Milne Cover Designer Anna Ishchenko Distributed to the book trade worldwide by Springer Science Business Media LLC 233 Spring Street 6th Floor New York 10013 Phone 1 800 SPRINGER fax 201 348 4505 e mail orders ny springer Sbm com or visit www springeronline com For information on translations please e mail rightsQapress com or visit www apress com Apress and friends of ED books may be purchased in bulk for academic corporate or promotional use eBook versions and licenses are also available for most titles For more information reference our Special Bulk Sales eBook Licensing web page at www apress com bulk sales The information in this book is distributed on an as is basis without warranty Although every precaution has been taken in the preparation of this work neither the author s nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the informati
173. OR Create trail on localhost og InDirectory ceesdrdet 7 With Prefix Andsize 8 5 Trail Access Treat as local file Use TCP IP Format Release Figure 10 19 Adding a trail file for the Data Pump process Using the Param File Editor remove the login information because you don t need it for this data pump and add the passthru parameter as shown in Figure 10 20 You can also add the mapping table here or use the Map for DPGGSI parameter interface to add it Param File Editor Figure 10 20 Data pump parameter file Stop the Extract EXTGGS1 wait for the lag to become zero and then stop the Replicat REPGGS1 In GGSCI change the Extract Replicat to point to the new trail files as shown here GGSCI alter extract dpggsi exttrailsource c ggs dirdat c1 EXTRACT altered GGSCI alter replicat repggsi exttrail c ggs dirdat c2 REPLICAT altered Note that the exttrail path must exactly match in both the Replicat and Extract For example dirdat c1 isn t equal to c ggs dirdat c1 from Director s perspective However the Oracle GoldenGate processes treat them the same and run them fine you just don t get the correct diagram as shown in Figure 10 21 205 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR 206 M y for EXTGGRI Trai C ggsidrdatiC Extract DPGGS fur ORNS Extract EXTGGS localhost gd localhost gg localhost og localhost calhost gg Data Source Lag
174. ORACLE SID as shown next In the example I chose instead to set the Oracle environment using the TNS alias of TargetDB instead of using SETENV You could also set ORACLE HOME and ORACLE SID prior to entering GGSCI and not use a TNS alias or SETENV Any of these methods will work as long as the variable values are valid SETENV ORACLE HOME u01 app oracle product 11 1 0 SETENV ORACLE SID TargetDB Next let s review the USERID parameter USERID In Chapter 2 you created the GoldenGate database user ID with appropriate security privileges Using the USERID parameter you can specify the GoldenGate database user ID and password to connect to the target database The GoldenGate password can also be encrypted as covered in Chapter 5 As mentioned earlier on the Extract for SQL Server you can use the SOURCEDB parameter with the USERID option SOURCEDB sqlserver userid sa PASSWORD userpw SOURCEDB specifies the SQL Server ODBC data source and USERID specifies the database User ID If you re using Windows authentication you can leave off the USERID and PASSWORD options ASSUMETARGETDEFS You can use the ASSUMETARGETDEFS parameter when the source and target tables are identical In the example the HR schema on the source and target is identical If the database tables being used on the source and target are different you need to generate a data definitions file instead Chapter 5 covers using a data definitions file MAP Y
175. Oracle GoldenGate To aid you in your journeys we have provided you with a list of technical resources and command tips to use as a reference guide as you work with the product The appendix consists of the following three sections e The first section provides you with a list of references to consult for further reading and investigation of Oracle GoldenGate topics discussed in the book e The second section provides you with a list of commonly used Oracle GoldenGate commands with syntax and an example of how to use each e third section contains a listing of useful Logdump commands and syntax to use for troubleshooting Oracle GoldenGate environments References for Further Reading The Oracle GoldenGate documentation set encompasses thousands of pages of documentation This section serves as a reference guide to the frequently used documents and support notes to help you delve deeper into the content The following installation and configuration guides for Oracle GoldenGate are available online via the Oracle Technology Network OTN site located at http otn oracle com Oracle GoldenGate SQL Server Installation and Setup Guide 11g Release 1 11 1 1 August 2010 Oracle GoldenGate MySQL Installation and Setup Guide 11g Release 1 11 1 1 October 2010 Oracle GoldenGate Teradata Installation and Setup Guide 11g Release 1 11 1 1 October 2010 291 www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACL
176. PENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR 296 Logdump Commands and Syntax for Troubleshooting The Oracle GoldenGate Logdump utility is the Swiss Army knife of analysis and troubleshooting You should understand and learn to use it because it s a potent tool in your arsenal By using Logdump you can view data in transactions present for a trail file identify missing transactions and troubleshoot replication issues such as duplicate and lost data transactions Let s look at some common techniques and commands to use with Logdump Accessing the Logdump Utility Logdump is an operating system utility that is accessed from outside the GGSCI interface To start a new Logdump session navigate to the Oracle GoldenGate base installation directory and type the command logdump as shown here C ggs mssqlserver gt logdump Oracle GoldenGate Log File Dump Utility Version 11 1 1 0 0 Build 078 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved Logdump 8 gt By default Logdump doesn t execute anything in and of itself It opens a unique prompt session that waits for your commands to perform tasks Getting Help with Logdump Syntax Logdump contains dozens of commands that present a challenge to new and even experienced Oracle GoldenGate administrators To obtain help with its cryptic syntax you can enter the help command as shown here Logdump 7 help FC num
177. PROPAGATIONDEL AY 100 THREADOPTIONS parameter 100 threads defining 101 102 increasing number of 180 thresholds maximum 155 Time Since Chkpt 156 TLTRACE parameter process tracing with 225 tnsnames ora file 100 102 TNSNAMES ORA file updating 102 103 tnsnames ora network configuration 220 TNSPING command 229 Tomcat Web Server Administration Tool screen 186 187 tools and utilities 45 47 DEFGEN 46 Director 46 47 GGSCI 46 Logdump 46 Reverse 46 Veridata 46 top command 130 top U gger 130 topologies 40 45 determining 281 of disaster recovery replication 243 one way replication 52 TOTALSONLY command 130 trace commands 225 226 process tracing with TLTRACE parameter 225 TRACE parameters 226 troubleshooting case study 226 TRACE parameters 226 TRACE2 parameter 226 TRACETABLE parameter 109 trail files changing size of 206 encrypting 93 94 generation rate of 160 162 missing 228 monitoring size 206 opening with Logdump utility 299 300 problems 221 223 not emptying 221 222 not rolling over 222 purging issues 222 223 purging old 89 90 287 sequence number 224 trails remote 38 39 source 36 TRANLOG parameter 63 www it ebooks info tranlogoptions extracting 206 207 TranLogOptions ASMUser parameter 102 transactions excluding for bidirectional replication 108 109 transfers of data 232 234 transforming columns 98 99 TRANSMEMORY parameter 2
178. PTRANSOPS setting of 1 000 SQL operations per Replicat transaction With BATCHSQL notice that GoldenGate changes the order of SQL operations and groups together the inserts updates and the deletes for the same SQL table BATCHSQL groups or batches together in a processing array the same SQL operations operating on the same table with the same columns 147 www it ebooks info CHAPTER 7 88 TUNING Remote Trail Replicat Transaction 1 Insert HR EMPLOYEE 1 Update HR EMPLOYEE 1 Delete HR EMPLOYEE 1 transaction Batch 1 Insert HR EMPLOYEE 1 Update HR EMPLOYEE 1 Delete HR EMPLOYEE 1 Batch 2 Insert HR EMPLOYEE 2 Update HR EMPLOYEE 2 Delete HR EMPLOYEE 2 Batch 3 Transaction 2 Insert HR EMPLOYEE 2 Update HR EMPLOYEE 2 Delete HR EMPLOYEE 2 Transaction 3 Insert HR EMPLOYEE 3 Insert HR EMPLOYEE 3 Update HR EMPLOYEE 3 Update HR EMPLOYEE 3 Delete HR EMPLOYEE 3 Delete HR EMPLOYEE 3 Figure 7 6 BATCHSOL GoldenGate transaction processing You may be asking yourself if using BATCHSOL causes any issues with referential integrity GoldenGate analyzes any foreign key dependencies before applying the batch and makes adjustments According to the GoldenGate documentation in some cases GoldenGate may need to apply more than one SQL statement per batch to maintain the referential integrity also creates a new section in your report file Here is an example of the new statistics y
179. Processing lag You can get the approximate lag from the info command and also the specific lag by running the lag command for each Extract and Replicat Server process statistics You can collect server statistics such as memory and CPU usage specifically for the Extract and Replicat processes For example on Linux if you have the top command available you can use a command like top U gger to monitor just the GoldenGate gger operating system user processes Database performance statistics part of your baseline you should include statistical reports on the source and target database usage such as an Oracle AWR report or a SQL Server dashboard report This gives you an indication of how much database activity is happening during your normal baseline Network performance statistics You should capture some statistics to indicate the network throughput and utilization You can use tools such as the Windows System Monitor or the netstat command on Linux to gather these statistics When you ve collected a solid set of baseline statistics you re well armed to deal with any performance issue that may arise Don t forget to update these baseline statistics periodically 130 www it ebooks info CHAPTER 7 88 TUNING The next question you re probably asking is what metrics should I collect and for how long This section offers guidelines for metrics you should collect in your performance baseline Keep in mind that these are only su
180. Product Release Platform T E name or select from list x amp Latest updates on Certifications 2 for Finding Certifications Fusion Middleware Certifications Enterprise Manager Certifications o t Product Availability Lifetime Support Watch a Video Tutorial z dassic Certify amp Product Roadmap Figure 6 1 My Oracle Support certification matrix Currently Oracle GoldenGate replication from Microsoft SQL Server to Oracle is supported with the versions shown in Table 6 1 Table 6 1 Supported Configurations for Oracle 11g GoldenGate with Microsoft SQL Server Source Source OS Target Target OS GG Version Microsoft SQL Microsoft Windows Server Oracle 9i Windows Linux Oracle 11g Server 2000 2000 2003 2008 10g 11g UNIX GoldenGate Microft SQL Microsoft Windows Server Oracle 9i Windows Linux Oracle 11g Server 2005 2000 2003 2008 10g 11g UNIX GoldenGate Microsoft SQL Microsoft Windows Server Oracle 9i Windows Linux Oracle 11g Server 2008 2000 2003 2008 10g 116 UNIX Goldengate 112 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION Preparing the Oracle GoldenGate Environments You will first need to install the Oracle 11g GoldenGate software on the source Oracle and target Microsoft SQL Server environments Be sure to install the sample database schemas and scripts included with GoldenGate and these database platforms Initial Data Load Completion for Oracle GoldenGate
181. R TABLE 57 AMERICAN AMERICA AL32UTF8 61 Apache Tomcat Web Server port 186 APPEND parameter 89 AQ Oracle Advanced Queuing 3 architecture 33 47 components 34 40 capture process 35 36 Collector process 38 301 www it ebooks info INDEX 302 data pump 36 38 databases 34 35 39 40 delivery 39 Local Extract process 35 36 Manager process 40 network 38 Replicat process 39 trails 36 39 tools and utilities 45 47 DEFGEN 46 Director 46 47 GGSCI 46 Logdump 46 Reverse 46 Veridata 46 topologies and use cases 40 45 typical flow 33 34 archived logs checking 157 158 archives database 235 ASM Automatic Storage Management configuration options for 102 103 specifying user 102 updating listener 102 updating TNSNAMES ORA file 102 103 ASMUser parameter 103 ASSUMETARGETDEFS parameter configuring for initial load Replicat file 73 for Replicat process 80 asterisk 268 attributes finding in Director application 209 210 automated key sequences 109 automatic processes startup and restart of 90 Automatic Storage Management See ASM configuration options for automation monitoring 162 169 disk space 168 169 lag scripts 163 167 memory and CPU scripts 168 process scripts 162 163 AUTORESTART parameter 90 287 AUTOSTART parameter 90 bad data 174 bad parameter errors 230 baselines of performance 129 131 basic replication 2 BATCHSQL parameter 146 149 BE
182. RSON o zj e t ming du Serch 3 News o Celebrity Video SB Messenger Games Fecebook Twetter E Technology Network Word ORACLE Ben Account Male fuge Out Products and Services Downloads Oracie Technology Network Store Support Education Partners About Communtion lams imis Technology Metwork Figure 2 5 OTN website Since OTN is a massive site it can be a little tricky to find where you actually obtain the Oracle Goldengate software Never fear when you scroll down the page you will find the list of documentation and software on the left side Figure 2 6 shows the links to software downloads and documentation available for Oracle GoldenGate S Oracle Technology Network World Essential Links About Us Become a Member Downloads Documentation Discussion Forums Product Technical Information Articles Topics Cloud Security and More Blog Developer amp Admin Newsletters Developer Day Workshops amp Events TechCast Live Oracle ACES Java net Learning Library Sample Code Books Oracle Magazine Top Downloads Get Java E Java Oracle Database 11g Release 2 Oracle Database XE Free Product Oracle Enterprise Manager 110 Oracle Fusion Middleware 11g including Oracle WebLogic Server Oracle JDeveloper 110 Free Product Oracle SQL Developer Free
183. Replicat make sure you restart them after any changes 203 www it ebooks info CHAPTER 10 88 GOLDENGATE DIRECTOR LLL LLL ST Ele Edit Actions Controls Help OQOSB OBA je as localhost og Map OG Extract EXTGGS pg Trail C ggsididatiCt Data Source Lag 00 00 00 localhost og Replicat REPGGS localhost gg Map for REPGGS i Lag 00 00 00 localhost 9g TP d Parameters Orade GoldenGate Log GGSCI My Alerts Disconnected Java GGSCI connected to localhost 9000 Copyright Ae 2003 2009 Oracle and or its affiliates All rights reserved GGSCI info all Program Status Lag Time Since Chkpt MANAGER RUNNING EXTRACT RUNNING 00 00 00 00 00 05 REPLICAT RUNNING PG 00 00 00 00 00 05 GGSCI Figure 10 18 Running the Extract and Replicat Adding a Data Pump Process It s a best practice to use an Oracle GoldenGate Data Pump process when two systems aren t on the same server so the GoldenGate process can survive a long outage without losing data To add a Data Pump process add a new Capture from the data source and name it DPGGSI Click OK Then add a trail file from the Extract DPGGSI object The prefix for the data pump trail file is C2 as shown in Figure 10 19 Trail Access is TCP IP now because Data Pump process sends the trail files directly to the server at the Replicat side 204 www it ebooks info CHAPTER 10 GOLDENGATE DIRECT
184. SERT INTO GGS GG LAG HISTORY AUDIT TIME LAG VALUES 5 L LAG COMMIT email alerts if the lag is more than 5 minutes IF L_LAG gt 300 THEN if you want Oracle to send email alert you can use this GGS SEND EMAIL shingchung aol com com 5555555555 vtext com Golden Gate Admin ggs somecompany com Golden Gate Data Is Behind Lag Is L_LAG Seconds END IF if you want use kick this off from shell script this dbms output will return the lag value DBMS OUTPUT PUT LINE L LAG EXCEPTION WHEN OTHERS THEN NULL END GET_LAG this is the send_email procedure for Oracle for your reference you can use your own CREATE OR REPLACE PROCEDURE GGS SEND EMAIL I TO EMAIL IN VARCHAR2 I FROM NAME IN VARCHAR2 I FROM EMAIL IN VARCHAR2 I SUBJECT IN VARCHAR2 I MESSEGE IN VARCHAR2 IS C UTL SMTP CONNECTION EMAIL START POS NUMBER 5 1 L EMAIL END POS NUMBER 5 0 L EMAIL CNT NUMBER 5 0 L EMAIL VARCHAR2 250 LAST EMAIL ADDR BOOLEAN FALSE PROCEDURE SEND HEADER NAME IN VARCHAR2 HEADER IN VARCHAR2 AS BEGIN UTL SMTP WRITE DATA C NAME HEADER UTL TCP CRLF END 165 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE BEGIN UTL SMTP OPEN CONNECTION your mail relay server UTL SMTP HELO C foo com UTL SMTP MAIL C I FROM EMAIL LOOP EMAIL L EMAIL 1 L EMAIL EN
185. SQL scripts to grant any necessary object privileges required by your application during a switchover or failover Processes and scripts to move your application and users from the source to the target server and back again First let s review the requirements for disaster recovery replication Understanding the requirements is probably the single most important factor in the success of any replication project Requirements You need to gain a solid understanding of the specific requirements which will drive the disaster recovery replication technical design Here are some of the typical requirements and considerations for disaster recovery replication We ll refer to the source database you are replicating from as the primary database and the remote target database as the standby or disaster recovery database The primary database also know as the trusted source is the active database where the application programs connect under normal circumstances Once you define your requirements you will need to ensure that you can meet these requirements using Oracle GoldenGate disaster recovery replication Keep the standby database synchronized with the primary database The primary database will be actively processing changes For the purposes of this chapter we will assume the standby database will be used for read only queries however remember that with GoldenGate you could allow writes to the standby database in a bidirectional confi
186. TA Improving Performance and Reducing Overhead Veridata processes can be very CPU intensive and sometimes I O intensive If you need Veridata to do a lot of work with a minimum footprint you should make every effort to reduce the system utilization to help the performance Because the default options are not necessarily the best options there are a few tricks that can improve performance and reduce unnecessary overhead Excluding Columns In Column Mapping you can use User Defined to remove the column s from the mapping The removed columns can be added back later on The primary keys can t be removed from the mapping as Veridata uses them to compare By default Veridata queries every column in both source and target to compare Tuning Profiles Settings There are two types of sorting methods database sorting and NSort on the Veridata Server If database servers have spare CPU or IO then use the default option of sort on database If the database is revenue generating environment that needs every single resource possible to get high Service Level Agreement SLA then we can usethe sortonserver option Please note Veridata server is also doing actual compares and other tasks if it peaks out 100 CPU during sort you may wish to sort the data on the database server or add more CPUs to the Veridata Server Disabling the Confirm Out of Sync Step If you don t expect the data to change during the compare such as daily weekl
187. TIONS Target HR LOCATIONS COLMAP USEDEFAULTS FILTER RANGE 1 3 www it ebooks info CHAPTER 7 88 TUNING Map HR DEPARTMENTS Target HR DEPARTMENTS COLMAP USEDEFAULTS FILTER RANGE 1 3 Map HR COUNTRIES Target HR COUNTRIES COLMAP USEDEFAULTS FILTER RANGE 1 3 Map HR REGIONS Target HR REGIONS COLMAP USEDEFAULTS FILTER RANGE 1 3 Add the parameter file for the new Replicat RHREMD2 GGSCI targetserver gt edit params RHREMD2 Replicat RHREMD2 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD2 dsc Append DiscardRollover at 02 00 ON SUNDAY Map HR EMPLOYEES Target HR EMPLOYEES COLMAP USEDEFAULTS FILTER GRANGE 2 3 Map HR JOBS Target HR JOBS COLMAP USEDEFAULTS FILTER 2 3 Map HR JOB HISTORY Target HR JOB HISTORY COLMAP USEDEFAULTS FILTER RANGE 2 3 Map HR LOCATIONS Target HR LOCATIONS COLMAP USEDEFAULTS FILTER QRANGE 2 3 Map HR DEPARTMENTS Target HR DEPARTMENTS COLMAP USEDEFAULTS FILTER QRANGE 2 3 Map HR COUNTRIES Target HR COUNTRIES COLMAP USEDEFAULTS FILTER GRANGE 2 3 Map HR REGIONS Target HR REGIONS COLMAP USEDEFAULTS FILTER QRANGE 2 3 Add the parameter file for the new Replicat RHREMD3 GGSCI targetserver edit params RHREMD3 Replicat RHREMD2 USERID GGER PASSWOR
188. UTF8 USERID GGER SourceDB PASSWORD userpw RmtHost targetserver mgrport 7840 RmtTask Replicat Group DHREMD1 Table HR One new parameter is required for the initial load Extract The RMTTASK parameter is used to have GoldenGate automatically start and stop a Replicat on the target server for the direct load During the direct load the Extract communicates directly with the initial load Replicat over the network Records are loaded by the Replicat directly into the target database as the Extract captures them from the source database The Replicat keyword tells GoldenGate that the remote task is a GoldenGate Replicat The Group parameter is used to specify the name of the Replicat on the target server in this case Replicat DHREMDI Adding the Initial Load Extract Now that you ve set up your initial load Extract configuration parameters the next step is to add the initial load Extract group You can do that using the following commands from GGSCI GGSCI sourceserver ADD EXTRACT IHREMD1 SOURCEISTABLE Notice that adding the initial load Extract is similar to adding the Local and data pump Extracts One difference is that you need to specify the SOURCEISTABLE option SOURCEISTABLE tells GoldenGate that this Extract is used only for initial loading and to capture all the records from the source database for loading into the target database Another difference is that there is no need to specify any trail files for the Oracle direct loa
189. VER parameter 89 REPORTRATE MIN command 130 REPORTROLLOVER parameter 89 ReportViewer role name Tomcat Web Server Administration Tool screen 187 repositories Veridata 173 requirements for disaster recovery replication 242 243 and planning 279 281 determining topology 281 knowing business objectives 279 280 understanding requirements 280 281 RESETREPORTSTATS option 89 Reverse utility 46 RHREMDI defs file 98 right trim RTRIM on character fields 181 RMTHOST parameter 230 231 configuring for data pump 67 tuning 151 RMTTASK parameter 71 RMTTRAIL parameter 67 230 role based security 174 186 188 row hash value 173 rows filtering 96 RTRIM right trim on character fields 181 Run Job button Veridata Main screen 179 RUN options altering for Extract and Replicat objects 206 Run Execute job tab Veridata Main screen 180 Running Jobs panel 176 180 RUNNING status 68 S SCOTT user schema 113 SCOTT DEPT table 113 SCOTT EMP table 113 scripts data 176 177 security encrypting 92 94 passwords 92 trail files 93 94 role based 186 188 Security requirements 50 281 Security tab 119 selective comparison 174 SEND command 79 295 SEND EXTRACT command 235 Seqno 155 157 158 161 157 158 160 161 Server and database component Director 190 www it ebooks info Server component Veridata 172 Server process statistics 130 servers
190. a Pack for 43M Microsoft Windows 32 bit Oracle Application Server 10g Release 10 1 3 0 0 B36235 45 NOV 01 2010 155 3 10 1 3 Media Pack for Microsoft 55G Windows Figure 2 2 Downloading Oracle Goldengate software from E Delivery Once you have chosen the correct Oracle Goldengate software which in our example is for the Windows platform you will be taken to the download screen shown in Figure 2 3 www it ebooks info CHAPTER 2 E INSTALLATION ORACLE E Delivery 3 Download Oracle GoldenGate on Oracle v11 1 1 0 0 Media Pack for Microsoft Windows 32 bit Back Search Again view the Readme file s to help decide which files you need to download Frequently Asked Questions How do I know which files Print this page with the list of downloadable files It contains a list of are required the part numbers and their corresponding description that you may What are the disk space need to reference during the installation process requirements How do I get my license code T Oracle GoldenGate on Oracle v11 1 1 0 0 Media Pack v1 for Microsoft More Windows 32 bit Readme View Digest Part Size Select Name ti mbor Bytes Download Oracle GoldenGate V11 1 1 0 0 for Oracle 10g on Windows XP 22232 01 10M 2003 2008 Download Oracle GoldenGate V11 1 1 0 0 for Oracle 11g on Windows 22233 01 10M 2003 2008 Total 2 Figure 2 3 E Delivery website for obtaining Ora
191. a pump reads the local 11 trail file and pumps it to the remote or target server as 12 Also notice that GoldenGate adds the same six character sequence number onto the trail file name starting with 12000000 then 12000001 then 12000002 and so on TABLE The TABLE parameter may look familiar by now This is the same TABLE parameter described for the Local Extract In the example you use wildcarding to pump all the tables in the HR schema HR to the target server If needed you coulf filter or transform data at this point using the data pump Instead you just pass on all the data that the Local Extract has captured Adding the Data Pump Now that you ve set up the data pump Extract configuration parameters the next step is to add the data pump Extract group You can do that using the commands shown in the following example GGSCI sourceserver gt ADD EXTRACT PHREMD1 EXTTRAILSOURCE dirdat 11 GGSCI sourceserver ADD RMTTRAIL dirdat 12 EXTRACT PHREMD1 MEGABYTES 100 Let s look at each of the commands in more detail The first command ADD EXTRACT adds the data pump Extract using the configuration parameters defined in the last section The EXTTRAILSOURCE parameter of the ADD EXTRACT command tells GoldenGate to use the trail file created by the Local Extract as the source for the data pump Extract In the Oracle example the 11 trail file is the source BEGIN NOW tells Extract to begin processing changes from the source trail as
192. a pump reads the records in the source trail and simply passes all of them on as is In GoldenGate terminology this is called passthru mode If data filtering or transformation is required it s a good idea to do this with the data pump to reduce the amount of data sent across the network www it ebooks info CHAPTER ARCHITECTURE Although technically a data pump isn t necessary you should usually configure a data pump anyway as good practice As mentioned earlier you could set up the Local Extract to send changes directly from the source server to the remote target without a data pump as shown in Figure 3 2 Source Capture Target Local Database atabase Extract Figure 3 2 GoldenGate flow without a data pump As you can see from Figure 3 2 in this configuration the Extract process can be directly affected by the network Adding a data pump to the configuration however introduces a layer to insulate the Local Extract from any disruptions caused by the network connection to the target or a problem with the target itself For example if there was a network issue between the source and the target this could cause the Local Extract to fail By having a data pump the Local Extract can continue to extract changes and only the data pump is impacted This way when the network issue is resolved the data pump can be restarted and will quickly process the previously queued changes from the source trail that the Loca
193. abase character set in the GoldenGate Extract or Replicat parameter file SETENV NLS LANG AMERICAN AMERICA AL32UTF8 Next we ll cover using naming standards for your GoldenGate components Developing Naming Standards Develop naming standards for your Extract and Replicat groups Doing so helps you identify at a glance what the Extract or Replicat is used for and where it s running Naming standards are especially important in problem solving situations where you need to get downed replication up and running again quickly The current limitation for Extract and Replicat group names is eight characters Here are some items you should include in the patterns you create for your naming standard The type of process such as Extract Data Pump or Replicat e The application that is using the process such as payroll sales or billing e The environment context where the process is running such as development or production e An indicator to show if there are multiple Extract or Replicat groups For example a Replicat group named RPAYRPI is the first Replicatfor the payroll application PAYR running in production You can easily tell RPAYRPI is a Replicat because the first character is an R The P1 at the end indicates the first Replicat running in the production environment Naming standards can also be developed for trail files although you re limited to only two characters You can use the first character to indicate whether a trail
194. abase checkpoint table Did you verify the source and target database character sets Are you following naming standards for your GoldenGate components Did you use a data pump e Management and monitoring Have you learned the GGSCI command shortcuts Did you use OBEY files for your commands Are you generating interim reporting statistics for your Extracts and Replicats Are you using a discard file Are you reporting on the health of the Extracts and Replicats Are you purging the old trail files to keep the disk from filling up Do you have the Extracts and Replicats set up to start automatically 289 www it ebooks info CHAPTER 14 TIPS AND TRICKS e Performance Did you run adequate performance tests Did you limit the number of local Extracts Are you using passthru mode for the data pumps Did you set up parallel Replicats if needed Did you use the fastest storage possible for your trail files Did you tune the underlying source and target databases You should review the checklist during your GoldenGate project Let your answers guide you to implement best practices that you may be missing out on Implementing those practices generally leads to an efficiently running GoldenGate environment 290 www it ebooks info APPENDIX Additional Technical Resources for the Oracle GoldenGate Administrator Now that you have read the book we hope that you now have a solid foundation in your quest to master
195. able and enable the triggers and cascade delete constraints Verifying the Manager Status You may remember from Chapter 2 and Chapter 3 the discussion of the GoldenGate Manager process which manages all of the GoldenGate processes and resources Before you can start the Extract and Replicat processes you need to verify that the GoldenGate Manager process is running on the source and target servers You can t start replication if the Manager isn t running For the basic replication configuration the Manager parameter file on the source and target servers needs to contain the port number as shown in the following example GGSCI server 1 edit params MGR port 7840 58 www it ebooks info CHAPTER 4 M BASIC REPLICATION Chapter 5 significantly expands on the use of Manager parameters but this is fine for now You can use the INFO MGR command to verify that the Manager is running GGSCI sourceserver 1 gt info manager Manager is running IP port sourceserver 7840 If the GoldenGate Manager isn t running you can start it using the following command GGSCI sourceserver 1 gt start manager Tip of the GoldenGate commands have abbreviations to save time and keystrokes For example instead of the command INFO MANAGER you can abbreviate it as INFO MGR The REPLICAT keyword can be abbreviated as REP and EXTRACT can be abbreviated as EXT Next let s move on to configuring the GoldenGate Local Extract Confi
196. ad Checkpoint File dirdat 12000003 2011 01 20 14 04 28 998416 RBA 3695 You can see from the INFO command that the Replicat RHREMDI is RUNNING Checkpoint lag is the time delay between the last checkpoint written to the trail and the time when the record was processed by GoldenGate You currently don t have any checkpoint lag If you had a high checkpoint lag it could indicate a performance problem or the Replicat catching up on a large volume of changes You can also see that trail file 12000003 is the source for your Replicat the last log read checkpoint time and the RBA of the record in the trail file GoldenGate also refers to the trail file number as the external sequence number EXTSEONO In this case EXTSEQNO is 3 You can start the GoldenGate Replicat processing at a specific EXTSEONO if needed You can add the DETAIL option to the INFO command to see even more information about the Replicat You may find the detail display helpful to see where the important files for the Replicat are located such as the parameter and report files The following example performs an INFO command the Replicat RHREMDI with the detail option GGSCI targetserver 3 info replicat RHREMD1 detail REPLICAT RHREMD1 Last Started 2011 01 22 22 40 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 07 ago Log Read Checkpoint File dirdat 12000003 2011 01 20 14 04 28 998416 RBA 3695 Extract Source Begin End dirdat 12000003 2011 01 20 13 57 2011
197. add trandata for the Extract tables To do so right click the data source again and choose View Details The dialog shown in Figure 10 12 opens Click the Tran Redo Log Data tab and then click Get Info Director inserts default login 200 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR information if you need other schema change the login information in the Database Access section and click Get Info to refresh the table list Highlight the tables and click the Add Trandata you can hold the Shift key to select multiple tables Details for localhost_gg x Report Info Definitions Files Tran Redo Log Data Checkpointing Use this panel to cause the primary key column s to be logged to the transaction log This is required for log based capture to make key values svsilable to GoldenGate for update operstions even though the database does not ordinarily log key values unless they are changed INSTALL ACNTPROPB INSTALL ACONPROCSB INSTALL ACONWATCHB INSTALL ALERTB NTALL ALITOINCR Status Logging supplemental is enabled INSTALL Logging supplemental i enabled INSTALL Logging supplemental is enabled INSTALL Logging supplemental is enabled INSTALL Logging supplemental i enabled INSTALL Figure 10 12 Adding trandata After you click Add Trandata you have the option to choose the trandata columns If you want to capture all columns select all the columns and click OK Now you
198. airs and one job has many groups For example you can break down the group by subject areas or by schedules Creating Compare Pairs GoldenGate Compare Pairs is what we need to compare We can compare tables materialized views and views This is the heart of Veridata where the business rules are It is part of the Group Configurations You can follow the steps below to set up the tables that we want to compare Ld A j a a Logout About ORACLE GoldenGate Veridata Home Group Configuration gt gt Edit Group group big table Running Jobs View By Job Selected Group View By Group View By Compare Pair Finished Jobs View By Job View By Group View By Compare Pair Nome group big table g Description Connection Configuration Group Configuration Tie conata Source Connection orci source i5 Run Execute Job Target Connection orci target i5 Go to Compare Pair Configuration Existing Compare Pairs Compare Pairs 1 amp Filters Enabled Name 4 Source Table Name Target Table Name page b group BIG TABLE SOURCE BIG TABLE TARGET BIG TABLE SOURCE BIG TABLE TARGET Showing From 1 To 1 Of 1 Compare Pairs Show 10 items per page M 4 M Figure 9 3 GoldenGate Group Configuration 177 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA Click the Go to Compare Pair Configuration link on the GoldenGate Group Configuratio
199. al design Here are some of the typical requirements As a reminder this section refers to the source database you re migrating from as the old database and the target database you re migrating to as the new database You need to ensure that you can meet these requirements using Oracle GoldenGate zero downtime migration replication e Keep the old database you re migrating from synchronized with the new database After the migration cutover keep the old database synchronized for some period of time until you re certain there will be no fallback to the old database e The old database will actively process SQL changes until the migration cutover Typically the cutover occurs during a scheduled weekend maintenance window or during a slow period for the website or application e During the interim period until the migration cutover the new database can be used for read only queries as needed Although supported by Oracle GoldenGate s bidirectional replication feature typically the data in the new database isn t being updated until after the cutover If updating data in the new database before the cutover is a requirement for your project you need to configure bidirectional replication to keep both databases synchronized 264 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION e Database data and structures in the old database can be different than the new database This depends on the type and complexity of migration Fo
200. al time replication data 182 tables and data scripts 176 177 components of 171 173 Vericom interface 173 Veridata Agent Java agent and C Code agent 173 Veridata Repository 173 Veridata Server 172 Veridata Web tool 173 Main screen 179 performance compare methods 180 comparing incremental data for large tables 181 182 INDEX disabling Confirm Out of Sync step 180 excluding columns 180 increasing number of threads 180 RTRIM on character fields 181 statistics 183 185 tuning profiles settings 180 role based security 186 188 Vericom interface command line 185 186 Veridata install dir server web conf directory 186 versions software 227 VIEW GGSEVT command 223 226 VIEW PARAMS group command 231 VIEW REPORT group command 235 VIEW REPORT command 219 w argument 186 W WARNLOGTRANS parameter 235 web client installation guidelines for Oracle GoldenGate Director 11g 26 Web component Director 191 web requirements Oracle GoldenGate Veridata 30 31 Web tool Veridata 173 WebLogic domain 190 331 www it ebooks info INDEX 332 WebLogic port 191 WHERE clause 96 181 WINIISRC environment 13 14 WIN11TRG environment 13 14 Windows Microsoft See Microsoft Windows Windows Event log 213 Windows event viewer 116 Windows Explorer graphical interface 229 Windows Service 115 wp argument 186 Write Checkpoint command 160 X Y
201. allowing both sides of the replication to be active the database and hardware can be fully utilized www it ebooks info CHAPTER 5 ADVANCED FEATURES Source Trail Replicat Remote a Data Pump Cg 3 Trail 0 1 iss Souce Taget 7777777 Source Target Database Local i Database 1Data Pump Extract 1 2 Source Trail Figure 5 3 BiDirectional replication GoldenGate configuration Following are some of the key components illustrated in Figure 5 3 e A local Extract running on each of the the source servers e pump running on each of the source servers This is optional but recommended The data pumps can also be moved to a middle tier server if needed to offload processing on the source server e Replicat running on each target Multiple parallel Replicats can be configured if needed to improve performance One drawback to bidirectional replication is that it can become extremely complex You must develop strategies for handling data collisions and avoiding key conflicts For example one strategy might be that each database can only process a certain range of key values to avoid collisions The following sections look at these topics and use the same sample HR EMPLOYEES table you ve been using throughout the chapter For bidirectional replication the HR EMPLOYEES table is both a source and a target on each database involved in the replication Before you start configuring th
202. an proceed with restarting the original replication from the old database to the new database in part 2 as shown in Figure 13 7 Start the Extract Start the Pump Start the Replicat 1 3 4 lt a Old Local E D gt New Database eal Data Pump Replicat Database e eR Source Remote Database Trail Trail Application Start the Application 2 Figure 13 7 Zero downtime migration fallback part 2 277 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION Following are descriptions for steps 1 4 as shown in Figure 13 7 1 To begin capturing changes in preparation for the application switchover back to the old database alter the Local Extract for the old database to begin capturing changes and then start the Extract as follows GGSCI oldserver gt ALTER EXTRACT LPREMP1 BEGIN NOW GGSCI oldserver gt START EXTRACT LPREMP1 2 Prepare the old database for the application and start the application connected to the old database In preparation you may need to grant some SQL permissions to your application users Also remember to enable any triggers or cascade delete constraints you may have stopped while the old database was the replication target 3 Startthe data pump Extract on the old database as shown in the following example GGSCI oldserver START EXTRACT PPREMP1 4 Startthe the Replicat on the new database Make sure to disable any triggers or cascade delete constrain
203. as as as as as as of of 2011 03 09 2011 03 09 2011 03 09 2011 03 09 2011 03 09 2011 03 09 2011 03 09 2011 03 09 2011 03 09 19 19 19 19 19 19 19 19 19 27 28 29 30 57 rate 32 33 34 06 rate 31 35 48 rate 50 rate 53 rate 55 rate 00 rate 02 rate 04 rate 4728 delta 4728 delta 3685 delta 3247 delta 2327 delta 1727 delta 1135 delta 3969 4439 4183 3366 2630 2257 1736 768 delta 1125 436 delta 923 In the example you re reporting on the Replicat processing rate every 10 000 records You can see from the Replicat report listing that the processing rate began decreasing at 19 28 and continued to decrease Review the GoldenGate ggserr 1og file and look for any errors or warnings Sometimes a warning message may be triggered that you may not be aware could be negatively impacting performance Review the server system logs for any error messages that could be contributing to the slow performance such as a bad disk drive Review the built in database performance reports such as AWR for Oracle or the SQL Server dashboard and compare them to the reports from the baseline period See if there are any significant changes that could be causing poor performance not know the exact solution yet but you should at least have some clear ideas to try to improve the performance The next step is to design and implement a solution www it ebooks info 13
204. as shown in Figure 4 1 You may remember from Chapter 3 that one way replication is the simplest topology and is often used for reporting or query offloading purposes Data is replicated from a single source database to a single target database in only one direction Changes to database data are only made at the source database and are then replicated to the target database Source Target Database Database Figure 4 1 One way replication Basic Replication Steps When the prerequisites are complete setting up basic one way replication can be accomplished in four steps as shown in Figure 4 2 These steps include initially loading the data and then keeping the data synchronized after the initial load The next sections cover each of these steps in detail www it ebooks info CHAPTER 4 M BASIC REPLICATION Performing the Initial Load 3 Starting the Extract Starting the Pump Starting Replicat Online D Local D Target Source Extract Database Database Source Trail Figure 4 2 Basic replication steps Following are descriptions of the steps in the figure 1 Start the Extract Configure and start the Extract to begin capturing database changes You need to start the Extract before the initial load so that any SQL data manipulation language DML changes made during the load process can be captured The Extract captures the database changes from the source database transaction log and writes th
205. ase GGSCI targetserver 2 gt add checkpointtable No checkpoint table specified using GLOBALS specification gger chkpt Successfully created checkpoint table GGER CHKPT Nowrun another checkpoint command to verify that the checkpoint table exists GGSCI targetserver 1 gt info checkpointtable gger chkpt Checkpoint table GGER CHKPT created 2011 01 30 22 02 03 Here are the basic Extract and Replicat parameter files along with the changes you ve introduced in this section shown in italics First let s look at the enhanced Extract parameter file GGSCI sourceserver 1 edit params LHREMD1 Extract LHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER SourceDB PASSWORD userpw ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt LHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY ExtTrail dirdat 11 Table HR 91 www it ebooks info CHAPTER 5 ADVANCED FEATURES 92 Next is the enhanced Replicat parameter file GGSCI targetserver 1 gt edit params RHREMD1 Replicat RHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER TargetDB PASSWORD userpw AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY Map HR Target HR Finally the enhanced Manager parameter file is shown here GGSCI server 1 gt edit params MGR
206. ase during your migration window Performing the Migration Cutover During the migration cutover you switch your database application from the old database to the new database After the application is cutover to the new database replication is reversed back to the old database to allow for fallback This section goes through the steps using Oracle GoldenGate replication to cutover to the new database To keep it simple the migration cutover is divided into two parts each with several steps see Figure 13 4 Let s begin with part 1 Stop the Extract Stop the Pump Stop the Replicat sum UR xm e NEM 2 tt Old Local ra D D New Database EA Data Pump X Replicat Database ES Source Remote Database Trail Trail Application Stop the Application 1 Figure 13 4 Zero downtime migration cutover part 1 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION Following are descriptions for steps 1 4 as shown in Figure 13 4 1 Stop the application connected to the old database 2 After verifying that there is no lag stop the Local Extract as shown in the following example GGSCI oldserver gt LAG EXTRACT LPREMP1 Sending GETLAG request to EXTRACT LPREMP1 Last record lag 10 seconds At EOF no more records to process GGSCI oldserver STOP EXTRACT LPREMP1 3 After verifying that there is no lag stop the data pump Extract as shown here GGSCI oldserver LAG EXTRACT PPREMP
207. ases 3 AReplicat is running on each target Multiple Replicats can be configured if needed to improve performance www it ebooks info CHAPTER ARCHITECTURE Integration Replication Integration replication as shown in Figure 3 11 is used to consolidate and integrate data from multiple source databases into a single target The target database is read only Integration replication is often used in data warehouse environments Data is usually transformed and filtered and only a subset of each of the source databases is needed for the data warehouse Source Database gt EN Target gt Database Source Database Figure 3 11 Integration replication A typical GoldenGate configuration to implement integration replication involves the components shown in figure 3 12 Source Remote Database Trail Replicat ene Target Database D Replicat D 3 Remote Database Trail Figure 3 12 Integration replication GoldenGate configuration Here is a more detailed description of each of the process components in the figure 1 A Local Extract is running on each of the source servers 2 A data pump is running on each source server each pumping data to the target database The data pumps can do transformation and filtering as needed 3 Two Replicats are running in parallel on the target one to process each trail file coming from the source 45 www it ebooks info CHAPT
208. base transactions 3 Startthe the Replicat on the primary database to prepare to process the changes that were made on the standby database as shown in the following example GGSCI primaryserver START REPLICAT RHREMD3 4 Startthe data pump Extract on the standby to send change transactions to be processed by the Replicat on the primary as shown in the following example GGSCI standbyserver START EXTRACT PHREMD3 In part you finished your database maintenance and then processed all the changes that were made to the standby database while it was acting as the primary database Next you need to turn off the replication from the standby database to the primary database in part 4 as shown in Figure 12 7 254 www it ebooks info CHAPTER 12 DISASTER RECOVERY REPLICATION Stop the Replicat Stop the Pump Stop the Extract C D G 2 1 reo a Local Btendhy Replicat E Data Pump Extract Primary seats Database Remote Source Trail Trail Figure 12 7 Disaster recovery planned switchover part 4 10 Following the descriptions of the steps for part 4 from Figure 12 7 1 After verifying there is no lag stop the Local Extract on the standby database as shown in the following example GGSCI standbyserver LAG EXTRACT LHREMD3 Sending GETLAG request to EXTRACT LHREMD3 Last record lag 5 seconds At EOF no more records to process GGSCI standbyserver STOP EXTRACT LHREMD3
209. ber of Compare Pairs 1 Number of Compare Pairs With Errors Number of Compare Pairs With 005 Number of Compare Pairs With 005 Number of Compare Pairs Cancelled Job Completion Status IN SYNC You can also redirect the output to a file and email to the monitoring team The w or wp arguments will return the correct Exit codes There are six exit statues from the GoldenGate manuals 0 The Vericom command finished successfully If w or wp was not specified the exit status is 0 if the job started successfully The actual compare errors will not be captured 1 Invalid vericom syntax was used 2 Vericom could not find the connection information by looking in the veridata loc or veridata cfg and the connection parameter url u lt url gt is not specified with the command arguments 3 Syntax is correct but missing required flags For example if j or c is specified then g lt group gt input is required 4 Job ran successfully but the compare status has something other than in sync 5 Communication error with Oracle Veridata Server For a full list of the Vericom command arguments type vericom helprun Setting Up Role Based Security Veridata role based security is very easy to set up It already defines a fixed number of roles You just need to add users or groups to the pre defined roles then Veridata will apply the security for you To configure the security role use the following link to g
210. ble that you use as your target table On the target database the table is named STAFF This section also discusses a few more differences Table STAFF Name Null Type EMPLOYEE ID PK NOT NULL NUMBER 6 FIRST NAME VARCHAR2 20 LAST NAME NOT NULL VARCHAR2 25 FULL NAME NOT NULL VARCHAR2 46 EMAIL NOT NULL VARCHAR2 25 PHONE NUMBER VARCHAR2 20 HIRE DATE NOT NULL DATE JOB ID NOT NULL VARCHAR2 10 WAGES NUMBER 8 2 COMMISSION PCT NUMBER 2 2 MANAGER ID NUMBER 6 DEPARTMENT ID NUMBER 4 Instead of using the column name SALARY as in the source database table the target database table has a column named WAGES You code the MAP parameter as shown in the following example Map HR EMPLOYEES Target HR STAFF COLMAP USEDEFAULTS WAGES SALARY Before you can start the replication you need to generate a GoldenGate data definitions file This is required because the source and target tables are now different Generating a Data Definitions File You must use a GoldenGate data definitions file whenever your source and target tables aren t considered identical by GoldenGate In the example there is one column with a different name so you must create a data definitions file The GoldenGate Replicat uses the data definitions file to map the source and target tables Because you re doing the mapping on the target server using the Replicat you generate the data definitions file on the source server and transfer it to the target server If
211. can create the Extract Right click the data source on which you want the Extract to run and select New Capture The only required field is Capture Name all the other fields are filled by Director Enter the Extract name EXTGGS1 Change the database access information to your own If your capture is on Oracle Real Application Clusters RAC you need to alter the Extract with the threads attribute as shown here in GGSCI 4 indicates four node RAC GGSCI alter extract extggs1 translog threads 4 begin now EXTRACT altered When the Extract EXTGGSI has been created right click the EXTGGSI icon in the diagram and select New Trail Enter the trail file options shown in Figure 10 13 and then click OK The diagram in your panel should look like the one in the Figure 10 14 201 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Create trail on localhost aa In Directory C gas Wirdat With Prefix Andsize wey 5 Trail Access Treat as local file Use TCP IP Format Release Figure 10 13 Add trail file options Extract EXTGGS Trail C ggs dirdatict localhost Dats Source N localhost aq localhost gg Lag 00 00 00 5 353333333334 3355333331 localhost ag Figure 10 14 Complete Extract tasks diagram Right click the Trail icon and select New Delivery Use REPGGSI as the Replicat name and enter the full path for the dirdat directory Click OK You now have a total of six icons in your
212. can name this replicat RHREMP2 Let s begin by taking a look at the Replicat parameters for RHREMP2 on the standby server 247 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION 248 GGSCI standbyserver 1 gt edit params RHREMP2 Replicat RHREMP2 USERID GGER PASSWORD userpw ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt RHREMP2 dsc Append DiscardRollover at 02 00 HandleCollisions should be turned off after the initial load synchronization HandleCollisions AssumeTargetDefs HR Target HR Now that you ve set up the Replicat configuration parameters the next step is to add the Replicat group In the following example you are adding the RHREMP2 replicat to the GoldenGate configuration The Replicat will process the 14 trail which was pumped over by the disaster recovery data pump GGSCI standbyserver ADD REPLICAT RHREMP2 EXTTRAIL dirdat l4 Now let s start the RHREMP2 replicat GGSCI standbyserver START REPLICAT RHREMP2 Once the Replicat has started you can verify it is running using the INFO REPLICAT command You should see a Status of RUNNING If you see a Status of STOPPED or ABENDED there may be a problem Let s do an INFO command on our Replicat RHREMPI to check the status as follows GGSCI standbyserver 2 INFO REPLICAT RHREMP2 You should also run the STATS command on your Replicat to make sure it is processing changes as in the following exam
213. can use tools such as the Windows System Monitor or the netstat command on Linux to gather network statistics You can compare these to the baseline network statistics to determine if you re experiencing an ongoing performance problem or just a temporary spike In addition to server commands you can also look at GoldenGate itself to see if network issues may be causing a performance bottleneck Let s walk through an example of how to detect a network bottleneck using GoldenGate commands Firstlet s run an info command on the Local Extract A portion of the output is shown in the following example GGSCI sourceserver info ext LHREMD1 showch 10 Write Checkpoint 1 GGS Log Trail Current Checkpoint current write position Sequence 11 RBA 977 Timestamp 2011 01 28 23 19 46 828700 Extract Trail dirdat 11 You should run the info command several times and compare the output specifically the values for the Write Checkpoint Look for the highest Write Checkpoint in each command output and make sure the RBA and timestamp are increasing Run the same commands for the data pump Extracts If the Write Checkpoint values are increasing for the Local Extract and notfor the data pump Extract then there is a network bottleneck for the data pump Extract The Local Extract is able to process data but the data pump is bottlenecked Eventually the data pump Extract backs up with transactions due to the network bottleneck and runs out of internal mem
214. case when new users perform Oracle GoldenGate configuration without sufficient knowledge of the product The following parameters are required for an Extract EXTRACT group name USERID ID PASSWORD pw RMTHOST hostname MGRPORT port RMTTRAIL trail name EXTTRAIL trail name RMTFILE filename EXTFILE filename TABLE source table 229 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 230 A Replicat requires the following parameters to perform online change synchronization REPLICAT group name SOURCEDEFS file name ASSUMETARGETDEFS USERID ID PASSWORD pw MAP source table TARGET target table If the configuration parameters for the Extract and Replicat are listed in wrong order then processing fails Oracle GoldenGate parameters are processed in the exact order listed in a parameter file Much like grammar rules in foreign languages you must use the correct order and syntax for parameter files Let s look at some key examples The parameter RMTHOST must precede the RMTTRAIL parameter The parameter TABLE or MAP must be listed after global and all the specific parameters that apply to it Syntax errors with parameter files are another source of mischief Oracle GoldenGate reports syntax problems in the process report and these often appear as bad parameter errors One way to mitigate these syntax errors which can be tricky to isolate in a parameter
215. cation you had to set up snapshot schemas and database links between the source and target database environments Data was mined from the online redo logs in Oracle to capture the changes and propagate them across the network to send data transactions from the source database a k a master to the target database The schemas had be configured on both the source and target databases as well In addition refresh groups had to be configured in order to keep the target environments in sync with the primary master database on the source system As you can imagine the design was crude and prone to errors in terms of establishing a successful conflict resolution system of checks and balances Oracle Advanced Replication Oracle advanced replication added more robust features to allow for multimaster replication with multiple environments as well as trigger based methods to establish conflict resolution rulesets to improve data reliability and to maintain accurate data synchronization between source and target www it ebooks info CHAPTER 1 INTRODUCTION systems Nonetheless advanced replication had its shortcomings and flaws particularly the lack of support for certain data types and failures that occurred with trigger based conflict handling Also latency was an issue in terms of keeping the target systems in sync with source master systems Figure 1 2 shows how conflict resolution was set up for basic and advanced replication in Oracle envi
216. cats 41 43 MySQL 5 x 122 MySQL RDBMS relational database management system to Oracle replication 122 MYSQLS environment 14 N naming standards developing 283 netstat command 130 150 NETSTAT command 232 network diagnostic tool 234 network performance statistics 130 network requirements 51 281 networks 38 configuration issues with 231 234 requirements for installing Oracle GoldenGate 17 tuning 150 151 New Data Source 120 318 new database capturing changes made to 275 next command Logdump 299 NLS_LANG variable 61 73 78 239 No Dynamic ports available error 234 nocompressupdates parameter 209 nodes synchronizing 100 NODYNAMICRESOLUTION parameter 156 NOMANAGESECONDARYTRUNCATIONP OINT parameter 62 NSort sorting method 180 OBEY command 56 obey filename command 63 81 OBEY files 285 286 Observer role in Director Admin Tool 193 obsolete Replicat group 222 OLIISRC environment 13 OL11TRG environment 13 one way replication 40 42 52 200 203 online redo logs current 157 OOS Out Of Syno report 173 OPEN command Logdump 299 operating systems configuration issues with 230 231 www it ebooks info requirements for installing Oracle GoldenGate 18 Oracle 11g network listener 220 Oracle Advanced Queuing AQ 3 Oracle e delivery downloading software from 5 8 Oracle GoldenGate Director 11g 24 27 Oracle GoldenGate Director Server ins
217. ceDB instead of using SETENV You can also set ORACLE HOME and ORACLE SID prior to entering GGSCI and not use a TNS alias or SETENV Any of these methods will work as long as the variable values are valid SETENV ORACLE HOME u01 app oracle product 11 1 0 SETENV ORACLE SID SourceDB For SQL Server you can use SETENV to specify the number of times GoldenGate will try to read the transaction log before abending the time to delay between retries SETENV GGS CacheRetryCount 25 SETENV GGS CacheRetryDelay 2000 The preceding example tells GoldenGate to try reading the SQL Server transaction log 25 times default is 10 and to wait 2000 milliseconds default is 1000 between each try before abending USERID In Chapter 2 you created the GoldenGate database user ID with appropriate security privileges Using the USERID parameter you can specify the GoldenGate database user ID and password to connect to the source database The GoldenGate password can also be encrypted as covered in Chapter 5 To connect to SQL Server you can use the SOURCEDB parameter with the USERID option as shown in the following example SOURCEDB sqlserver userid sa PASSWORD userpw SOURCEDB specifies the SQL Server ODBC data source and USERID specifies the database user ID If you re using Windows authentication you can leave off the USERID and PASSWORD options Tip The order of the Extract parameters listed is important For example the EXTRA
218. changes from the PAYROLL database except for the two tables you re excluding Let s look at the Replicat parameters for RPREMPI on the new server as shown in the following example GGSCI newserver 1 edit params RPREMP1 Replicat RPREMP1 USERID GGER PASSWORD userpw ReportCount Every 10000 Records Rate Report at 00 30 270 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION DiscardFile dirrpt RPREMP1 dsc Append DiscardRollover at 02 00 HandleCollisions should be turned off after the initial load synchronization HandleCollisions AssumeTargetDefs Map PAYROLL Target PAYROLL You may wonder why you don t exclude the two tables that you excluded in the Extracts The reason is that the tables were already excluded in the Extracts so the data isn t included in the trail files being processed by the Replicat You can simply tell Oracle GoldenGate to apply all the data in the trail file for the PAYROLL database which doesn t include the two excluded tables Now that you ve set up the Replicat configuration parameters the next step is to add the Replicat group The following example adds the RPREMPI Replicat to the Oracle GoldenGate configuration GGSCI newserver ADD REPLICAT RPREMP1 EXTTRAIL dirdat p2 Tip When adding your Replicat you can use a specific checkpoint table name in the database or use the default database checkpoint table specified in the GLOBALS file as discussed i
219. chapter covered the GoldenGate bidirectional replication topology setup The next chapter expands even further on replication scenarios by configuring GoldenGate for heterogeneous replication 110 www it ebooks info 6 Heterogeneous Replication In previous chapters we discussed how to install and configure basic replication for an Oracle 11g GoldenGate environment Based on experiences in the field at customer sites most implementations for Oracle 11g GoldenGate involve complex heterogeneous environments with multiple platforms and database vendors For instance while implementing a recent configuration for a large telecommunications customer we had to install and configure Oracle GoldenGate to replicate data from a Microsoft SQL Server 2005 data warehouse running under Microsoft Windows 2003 Server to an Oracle 11g data warehouse system on Linux Before Oracle GoldenGate existed data replication between different platforms and RDBMS systems was problematic at best and nigh well impossible at worst It involved hundreds of man hours to create custom scripts and development to take advantage of poorly documented system level APIs and even then it was a crapshoot Well fret no more With Oracle GoldenGate the challenge of replicating data in real time between different platforms and databases is a cake walk by comparison It is more difficult than basic replication setups for Oracle GoldenGate that only require that the data b
220. cle Goldengate software At this point we recommend that you review the readme file and release notes to best prepare for the installation process By investing a couple of hours you will avoid most errors during the installation and configuration phase Since we will use Oracle 11g with Goldengate we will choose the Oracle GoldenGate 11 1 1 0 0 for Oracle 11g on Windows XP software to download Note If you plan to use Oracle GoldenGate with Linux you will also need to download the Oracle GoldenGate software for the Linux platform Figure 2 4 provides the readme instructions for the Oracle GoldenGate media pack www it ebooks info CHAPTER 2 88 INSTALLATION Readme for Media Pack B59977 01 Print Close INSTRUCTIONS This table can help you determine what downloads are required to run your licenses Additional Oracle product downloads for which you have not purchased a license may be used only on a trial basis Please identify the license and options you would like to run and correspond them to the files that you have to download For information on the licenses that you have ordered please refer to your Oracle Ordering Document For information on trial license terms and conditions please refer to the E Delivery Trial License Agreement See Legend below on details regarding table notation X Required downloads in order to run the associated products and licenses D Documentation C Download the Client software if applicable
221. cluster you should capture new baseline metrics because the old metrics are no longer valid 129 www it ebooks info CHAPTER 7 88 TUNING Let s look at a sample of the metrics you should collect for your performance baseline Server statistics For the servers where you have replication running you should collect CPU utilization disk utilization and memory usage How you collect these statistics varies by platform For example on Linux you may use SAR or IOSTAT and on Windows you can use perfmon You may even have your own custom tools to collect these metrics Extract and Replicat statistics For each Extract and Replicat you should capture the following metrics Name and status of each processing group You can capture the output of the GoldenGate Software Command Interface GGSCI info command e Processing statistics You can use the stats command and get an aggregate processing rate for your Extracts and Replicats using a command such as STATS RHREMD1 TOTALSONLY REPORTRATE MIN You can use this command for example to get the total operations per minute processed by the Replicat Processing rate Chapter 5 showed how to add reporting parameters to the Extract and Replicat to automatically report performance statistics You can now view the report file for each Extract and Replicat and record the statistics From the report file you can record the number of records processed the rate and the delta
222. compared with the current statistics Determine if there are significant differences In the example you can see that the CPU utilization on the target server is significantly higher You can also see that the Replicat processing rate has dropped and the lag has increased significantly This points you in the direction of a possible problem with the Replicat but it could be something on the target database or the server or even something else 2 Conducta short meeting with the stakeholders to determine their perspective on what the current performance problems are and establish their expectations for tuning results It s important to understand what they believe is their most significant problem You can also share the initial findings from your comparison of baseline to current performance statistics Here are some examples of questions you should ask during the meeting e What are the most significant performance problems What business processes being impacted What time of day does the performance problem occur or is it constant 132 www it ebooks info RHREMD1 RHREMD1 RHREMD1 RHREMD1 RHREMD1 RHREMD1 RHREMD1 RHREMD1 When you ve finished analyzing the statistics stakeholder communication and information about the performance problem you should have a pretty good idea of what is causing the problem You may Does the performance problem affect only one user or all users CHAPTER 7 TUNING Is the perf
223. compression ratios of 4 1 or greater are possible Keep in mind the compress and uncompress add some CPU overhead to your servers 151 www it ebooks info CHAPTER 7 88 TUNING 152 Tuning the Database This book doesn t spend a great deal of time covering general database tuning but it s safe to say a poorly tuned database will negatively impact GoldenGate performance This is especially true on the Replicat side where GoldenGate is executing actual database DML statements to change the data You should ensure that your database is tuned properly prior to turning on GoldenGate There are a few major areas you can check to help ensure proper Replicat performance on the target database e Verify that the target tables used by GoldenGate have unique keys and SQL indexes are created on those keys e Verify that database optimizer statistics are kept up to date to ensure that the database chooses the most efficient SOL execution plans e sure the database tables that are the target of replication aren t overly fragmented If so they should be reorganized to eliminate the fragmentation e database performance reports such as AWR for Oracle database and verify there are no major performance issues Summary This chapter covered a basic methodology for tuning GoldenGate replication It reviewed capturing baselines of GoldenGate performance statistics to help you determine when you have a performance issue and how it can
224. configuration Enhancing Extract and Replicat Reporting By default the Extracts and Replicats generate a standard startup report every time they re started The startup report contains a lot of useful information about the Extract or Replicat such as the operating system database version character sets and parameters used For an example let s look at the report for the human resources or HR schema Replicat you set up in Chapter 4 named RHREMDI AK A A AK A OK OK KK OK KK KK Oracle GoldenGate Delivery for Oracle Version 11 1 1 0 0 Build 078 Linux x64 64bit optimized Oracle 10 on Jul 28 2010 15 58 11 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved Starting at 2011 01 28 23 19 11 RR K KK K Kok K KK K K K K K K K K K K K K K K K K K K K K K K K KK KK K KK K KKK KKK KK KKK Operating System Version Linux Version 1 SMP Tue Jan 23 12 49 51 EST 2007 Release 2 6 9 42 0 8 ELsmp Node targetserver Machine x86_64 soft limit hard limit Address Space Size unlimited unlimited Heap Size unlimited unlimited File Size unlimited unlimited CPU Time unlimited unlimited Process id 15874 86 www it ebooks info CHAPTER 5 ADVANCED FEATURES Description AK KKK AK KKK AK KK KKK KK
225. configuration for ASM 102 103 of character sets issues with 239 DDL replication 103 106 issues with 226 234 database availability 227 228 incorrect software versions 227 missing process groups 228 missing trail files 228 with network 231 232 with operating system 230 231 with parameter files 229 230 for RAC 100 102 config xml file 191 Confirm Out of Sync step disabling 180 confirmation compare 173 Confirmation Compares profile setting 180 Confirmed Out Of Sync COOS reports 174 conflict resolution handling for bidirectional replication 109 110 connection configuration 176 Connection Configuration option 176 connections database 175 176 constraints cascade delete disabling 57 58 INDEX key tables missing 239 consumption rate of redo logs 158 159 convchk utility 237 238 COOS Confirmed Out Of Sync reports 174 corrupted checkpoint errors 237 CPU Central Processing Unit scripts monitoring 168 CREATE TABLE AS SELECT option 113 credentials database privileges and 27 cross database comparison 174 CSI Customer Support Identifier account 292 CUSTOMER table 288 cutovers migration 274 276 D data synchronization issues 238 239 transfer issues with 232 234 Data currency 50 128 280 Data Definition Language DDL replication 103 106 data definitions files generating 97 98 data filtering columns 95 96 rows 96 tables 94 95 305 ww
226. ctor behind the scenes It s out of this chapter s scope to go over every scenario but you should be able to use Director to do the following additional tasks e Perform DDL synchronization with the DDL and DDLOPTIONS parameters e Perform Active Active synchronization with collision handling using the IGNOREREPLICATES GETAPPLOPS and TRANLOGOPTIONS EXCLUDEUSER parameters e Change log captures with INSERTALLRECORDS and GETUPDATEBEFORES e Troubleshoot and fix most common Oracle GoldenGate errors using Director Client 215 www it ebooks info 11 Troubleshooting Oracle GoldenGate As database professionals one of the most difficult aspects of working with Oracle GoldenGate is how to troubleshoot and resolve errors that occur throughout the lifecycle of the implementation Due to the nature of Oracle GoldenGate functionality and operations as enterprise software it involves all layers of the technology stack from source to target database to network issues and more The goal of this chapter is to provide you with the best possible overall framework to identify and resolve problems that occur within the Oracle GoldenGate ecosystem Common Problems and Solutions In order to equip you with the best possible approach to solving issues that arise with Oracle GoldenGate you look first into the most commonly found problems by using a holistic approach to troubleshooting Oracle GoldenGate environments Because Oracle Go
227. d Extract Before you start the initial load Extract you need to configure and add the initial load Replicat 71 www it ebooks info CHAPTER 4 M BASIC REPLICATION 72 Configuring the Initial Load Replicat Next let s examine the Oracle direct load Replicat parameter file used for the example Once again you re loading all the records in the HR schema from the source database into the same schema on the target database The Replicat is started and stopped automatically by GoldenGate when you start the initial load Extract GGSCI targetserver 1 edit params DHREMD1 Replicat DHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER TargetDB PASSWORD userpw AssumeTargetDefs Map HR Target HR Let s look at each of the Replicat parameters in more detail REPLICAT You can use the REPLICAT parameter to define the GoldenGate initial load Replicat group You should develop naming standards for your Replicat groups This will help you quickly identify what the Replicat is used for and where it s running This is especially important in problem solving situations where you need to quickly get the replication up and running again if it s down The current limit for Replicat group names is eight characters Here are some items you should include in your naming standard e The type of process in this case a Replicat As shown in the example you should also name initial load Extracts and Replicats differently if you
228. d Linux platforms be sure to verify that both the source and target Oracle database servers are online and available This means that in addition to checking to ensure that these Oracle instances and databases are online you need to also test network connectivity between both server hosts If Goldengate cannot access the hosts via TCP IP network connection the software will not function correctly You can use the ping network utility to check for network connectivity In the event that a network timeout problem exists between the source and target environments you will need to work with your network administrator to address this issue before installing the Oracle Goldengate software General System Requirements The basic installation requirements using Oracle with Goldengate are standard across Windows Linux and UNIX platforms in terms of supported Oracle database versions memory requirements disk space allocation network requirements and operating system support We cover all of those areas in this section Database Server Versions The following Oracle database versions are supported for Oracle 11g Goldengate e Oracle 9iR2 9 2 e Oracle 10gR1 R2 10 1 10 2 e Oracle 11gR1 R2 11 1 11 2 As of Oracle Goldengate 11g both DML and DDL replication are supported for the above Oracle releases Support for legacy versions of Oracle such as Oracle v7 and Oracle 8i is not provided We recommend that you upgrade your database to Oracle 115 if y
229. d and login issues on the source Oracle database you need to stop both the Manager and Extract processes with Oracle GoldenGate Exit the GGSCI interface for Oracle GoldenGate and then restart the Manager and Extract process groups on the source system 235 www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE 236 Data Pump Errors Recall from earlier chapters that you can use a special Extract process group called a data pump on an intermediate system to enhance performance and availability with processing One common problem occurs when you use the PASSTHRU parameter and try to use data transformations This causes the data pump to fail because the PASSTHRU parameter isn t supported with Oracle GoldenGate For the data pump pass through mode with Oracle GoldenGate both source and target table names and structures must be the same and no filtering operations can be performed on the data Another thing to check for in the data pump parameter file when you use the PASSTHRU parameter is whether you re using the USERID or SOURCEDB parameter If the source system doesn t contain a database these parameters shouldn t be used If you re using pass through for some tables and normal processing for other tables then for normal processing of these tables the system must have a database In addition you need to specify the database login parameters with Oracle GoldenGate as well as use a source definitions file if any filtering is per
230. d for running Oracle Goldengate on the Windows platform Requirements for Microsoft Windows Clustered Environments Goldengate has some unique requirements that apply to Windows environments using Microsoft Clusters Execute the following steps before performing your install 1 2 As administrator privileged account log on to one of the cluster nodes Select a drive that has resource membership within the same Microsoft Windows Cluster group Make sure that the disk group has cluster node ownership Place the Oracle Goldengate software on a shared drive accessible by all nodes in the cluster Installing Goldengate on Windows Following is a complete run through of a Goldengate install onto a Windows system The process begins with downloading the software continues through uncompressing the download file and then moves into the install proper 1 2 18 Download the Oracle 11g Goldengate software for Windows x86 platform Download and install the Microsoft Visual 2005 SP1 Redistributable Package from http microsoft com as shown in Figure 2 11 www it ebooks info CHAPTER 2 88 INSTALLATION Microsoft Visual 2005 SP1 Redistributable Package x86 Brief Description On this page The Microsoft Visual C 2005 SP 1 Redistributable Package x86 installs Quick Details runtime components of Visual C Libraries required to run applications developed with Visual C a computer that
231. d to do anything during the installation process Director Repository database stores all the configurations and the statistics in clear text format so we can query the tables for our ad hoc reporting 190 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Director Web Director Web monitors and controls the Oracle GoldenGate Extract and Replicat processes It s usually used by the production support team that is monitoring the software Director Web allows you to start and stop the process but you can t change the parameter files so you can t do advanced programming with Director Web It has a limited GGSCI interface that you can use to manage the Extract and Replicat processes Refer to the Oracle GoldenGate Administrator s Guide for the supported GGSCI commands To use Director Web type this URL in your web browser http dirhost abc com 7001 acon Change dirhost abc com to your Director Server IP or hostname 7001 is default WebLogic port to change the default port to something else you need to change the config xm1 file in the Director domain lt listen port gt 80 lt listen port gt lt listen port enabled gt true lt listen port enabled gt With Director Web you can even administer and monitor Oracle GoldenGate tasks from any compatible browser including Blackberry iPhone and iPad as shown in Figure 10 2 12 09 PM GoldenGate Director Web 4 Ir Co gm 192 68 1 7 7001 acon j
232. d to have the following prerequisites in place before you can start setting up replication for zero downtime migration e The Oracle GoldenGate software installed on the source and target servers as described in Chapter 2 e The Oracle GoldenGate database User ID created on the source and target databases 263 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION e server name or IP address of the target database server e The Oracle GoldenGate Manager process up and running on the source and the target e TCP IP network connectivity open from the source server to the target server s Oracle GoldenGate manager port and visa versa Anunderstanding of the business and technical replication requirements of the planned migration A few additional prerequisites are important for zero downtime migration replication e Backups of your Oracle GoldenGate software and working directories on the source and target servers e SQLscripts to grant any necessary object privileges required by your application during the cutover or fallback procedure e Processes and scripts to move your application and users from the old to the new database and back again Let s start by reviewing the requirements for zero downtime migration replication Requirements Before you begin to configure the replication you must have a solid understanding of the specific requirements that drive the zero downtime migration replication technic
233. d to improve performance 42 www it ebooks info CHAPTER ARCHITECTURE Bidirectional Replication In bidirectional replication as shown in Figure 3 7 changes to data can occur on either database at the same time and are replicated to the other database Each database contains an identical set of data Bidirectional replication is sometimes called active active replication because each side of the replication is actively processing data changes Source Target Source Target Database Database Figure 3 7 Bidirectional replication Figure 3 8 illustrates a typical bidirectional configuration Such a configuration is often used in environments where there are high volume and high performance requirements By allowing both sides of the replication to be active the database and hardware can be fully utilized Source Trail 1 Data Pump eke Souce Taget J WQS Source Target Database p Local Database Extract 1 Source Trail Figure 3 8 Bidirectional replication GoldenGate configuration Following are some of the key components illustrated in Figure 3 8 1 ALocalExtract is running on each of the source servers 2 data pump is running on each of the source servers This is optional but recommended The data pumps can also be moved to a middle tier server if needed to offload processing on the source server 3 AReplicat is running on each target Multiple parallel Replicats can be configu
234. d to log in to all the servers You only need to log in to the Director Client then you can administer all the Oracle GoldenGate servers that are set up in the Director repository GoldenGate developers administrators don t even need to know the UNIX Windows login Not having to log in to each system is a very nice feature if your GoldenGate servers are on different platforms and databases Director handles the database versions and platform specific commands and parameters for you The GoldenGate Software Command Interface GGSCI command line doesn t have an automated monitoring and alert feature Director displays the near real time lag on each process You can also set up lag and abend reports in a few minutes without writing any custom scripts With all these features Director enables rapid development a faster learning curve and quicker testing for most Oracle GoldenGate tasks It only takes two minutes to set up an initial load task and about five minutes to set up a simple Extract and Replicat This chapter goes over the key components the Director Then you walk through the process of setting up the initial load one way replication and the data pump process At the end the chapter discusses some of Director s advanced features Director Components Director consists of five components as shown in Figure 10 1 They are GoldenGate GGSCI instances that have Manager processes running Director Administrator that is used to configure
235. data is covered in detail in Chapter 9 Veridata Director Another add on product available from GoldenGate is Director Director is a graphical tool that you can use to centrally design configure manage and monitor your entire GoldenGate environment Director is made up of several components including a centralized server and database repository a client application and a web browser interface Director can also be integrated with third party monitoring solutions Director can manage all the databases and platforms that GoldenGate replication supports It s covered in Chapter 10 Director Summary In this chapter you learned about the GoldenGate replication flow and each of the components in detail It s important to understand the components because almost all GoldenGate scenarios use these same components You also saw some typical replication topologies and how they can be implemented using GoldenGate Specifically you learned about the topologies listed in Table 3 3 As you move into the next chapters you learn how to apply the GoldenGate architecture principles to implement GoldenGate replication Table 3 3 GoldenGate Topologies and Use Cases Topology Cardinality Data State Data Changes Use Cases One way One to one Active Passive Source only Reporting query offloading hot standby Bidirectional One to one Active Active Sourc eand High volume high performance target Broadcast One to Active Passive Source onl
236. denGate 11 data types such as BFILE and ANYDATA aren t supported Other data types may be supported but with certain limitations Be sure to check the Oracle GoldenGate product documentation for a list of data types supported 280 www it ebooks info CHAPTER 14 8 TIPS AND TRICKS e Security requirements How much security is required For example is it necessary to encrypt the data GoldenGate is replicating GoldenGate can encrypt both the data stored in trail files as well as the data being sent from the source to the target Also is the data in the database encrypted For example Oracle can encrypt database data using a feature called Transparent Data Encryption TDE If the database is using TDE GoldenGate needs special parameters to unencrypt the data for replication Finally GoldenGate itself requires security to install and own the GoldenGate software and execute the replication inside the database You need to plan for this and ensure that your organization allows the new users and security access required for GoldenGate e Network requirements Describe the network between the source and the target Are there any firewalls that need to be opened for the replication What is the distance How much bandwidth is required for the replication and how much is available Determining the Topology As discussed in Chapter 3 Architecture GoldenGate supports many different topologies Each topology has its own unique requireme
237. denGate v11 1 1 0 0 for Oracle 11 on Linux x86 64 61 MB Beehive Oracle GoldenGate v11 1 1 0 0 for Oracle 10g 64 bit on Windows XP 2003 2008 13 MB Oracle GoldenGate v11 1 1 0 0 for Oracle 11 64 bit on Windows XP 2003 2008 13 MB Application Server BPEL Process Manager Business Intearation Management Pack for Oracle GoldenGate Figure 2 9 Download Oracle Goldengate from OTN website Now that you understand how to obtain and download the Oracle Goldengate software and documentation it s time for us to perform the installation and configuration for our test environments Understanding Your Environment Before installing you should have a good grasp of your target environment For example you may want to install both Linux and Windows for the Oracle 11g database environments In Table 2 1 we show a test configuration that can be used for working with Oracle GoldenGate as a sandbox environment We also recommend that you install the sample schemas included with Oracle 11g to have sample test data while working with GoldenGate Our test platform will include the configuration in Table 2 1 www it ebooks info CHAPTER 2 88 INSTALLATION Table 2 1 Windows Test Environment for Oracle 11g GoldenGate for Homogeneous Replication Environment OS Database Software Function WIN11SRC Windows Oracle Oracle 11g GoldenGate Source System XP 11 2 v11 1 1 0 WIN11TRG Windows Oracle Oracle 11gGoldenGate Target System XP 11 2 v11 1
238. detail It also looks at some of the different replication topologies supported by GoldenGate Finally you take a quick glance at some of the GoldenGate tools and utilities You should keep some key GoldenGate architectural concepts in mind from the start the GoldenGate architecture is non invasive modular and flexible and it can support many different replication scenarios and use cases GoldenGate components can be rearranged and scaled up or down to meet specific replication scenario requirements And GoldenGate can replicate high volumes of data in real time with low server and database impact and never miss or duplicate a transaction Typical GoldenGate Flow Most replication scenarios involve at a minimum the components discussed in this chapter Figure 3 1 shows how these components combine to form what this book refers to as the typical GoldenGate flow Capture j 1 i P Source pa D Deliver Target Database gt Local i Replicat Database 1 Extract i UI E S E Source pa r4 1 Collector Trail i pa Remote Trail Figure 3 1 The typical GoldenGate flow 33 www it ebooks info CHAPTER ARCHITECTURE 34 The typical GoldenGate flow shows new and changed database data being captured from the source database The captured data is written to a file called the source trail The trail is then read by a data pump sent across the network and written to a remote trail file by the
239. dirdat p1 GGSCI oldserver ADD RMTTRAIL dirdat p2 EXTRACT PPREMP1 MEGABYTES 500 After adding the data pump Extract you need to start it to begin processing records from the source trail file GGSCI oldserver START EXTRACT PPREMP1 Next you can verify that the data pump Extract has started using the INFO EXTRACT command GGSCI oldserver 2 INFO EXTRACT PPREMP1 269 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION You should see a Status of RUNNING If you see a Status of STOPPED or ABENDED there may be a problem You should also run the STATS command on the data pump Extract to confirm it s processing data GGSCI oldserver 2 gt STATS EXT PPREMP1 Now that the Local Extract and Data pump Extract are started you can begin the initial data load You learned about the different methods for initially loading data in Chapter 4 These methods also work for the initial data load for the zero downtime migration replication Remember that the initial load process is happening during the period prior to the actual application migration cutover This period can be a few days weeks or even months before the actual cutover You can perform the initial load and have the data synchronized waiting for the scheduled cutover window You can use either Oracle GoldenGate itself or the DBMS vendor load utilities to do the initial data load Refer to Chapter 4 for more details on the initial load process Configuring
240. does not have Visual C EAD T 2005 installed System Requirements File Name Size vcredist x86 exe 2 6 MB Figure 2 11 Microsoft Visual package for Oracle GoldenGate installation on Windows platform After you have downloaded the Microsoft Visual package save the file and then click the vcredist x86 exe file as shown in Figure 2 12 You have chosen to open vcredist x86 exe which is a Binary File from http download microsoft com Would you like to save this file i HE Figure 2 12 Installing the Microsoft Visuall C libraries 3 Create two new directories for the Oracle Goldengate software to be the source directory and the other to be the target mkdir ggs src mkdir ggs trgt 4 Using Winzip or your favorite extraction utility extract the Oracle 11g Goldengate software to the C ggs_src directory on the source database server and to the C ggs_trgt directory to the target database server 19 www it ebooks info CHAPTER 2 INSTALLATION 20 5 Execute the command ggsci on source and target ggsci 6 Execute the command CREATE SUBDIRS on source and target create subdirs 7 Finally give the command EXIT to leave the ggsci environment to complete the installation process Do this on both the source and the target Imachines Installing GoldenGate 11g on Linux and UNIX We recommend that you have a basic understanding of Linux and UNIX comman
241. ds before you undertake an installation for Oracle Goldengate on Linux and UNIX platforms Basic syntax and knowledge of copy and archive commands such as cp mv tar and mkdir are essential to function within this environment Oracle ACE Director Arup Nanda has written an excellent tutorial series on Oracle Linux command mastery available online from OTN for Linux and UNIX novices and newbies at www oracle com technetwork articles linux index html Note Make sure that all system network and database requirements have been completed You will need to unzip the files for Linux and UNIX platforms by using the tar command Once you have unzipped the files run the ggsci command CREATE SUBDIRS as we performed previously for Windows Environment Settings for Oracle and Goldengate on Linux and UNIX Since Linux and UNIX platforms use environment variables to configure software you must set the ORACLE HOME and ORACLE SID environment variables For multiple process groups with Extract and Replicat you will need to configure the parameter files by adding SETENV as shown in the following syntax for the extract and replicat parameter files SETENV ORACLE HOME ORACLE HOME directory path SETENV ORACLE SID ORACLE SID If you have more than one Oracle database instance on a system with Extract and Replicat processes you will need to issue a SETENV statement in the parameter file for each Extract and Replicat process group Follow
242. e database A can have an additional key column to indicate that the record belongs to database A Database B can have a similar key added You need to evaluate any disadvantages of having such a column in your database tables Automated key sequences In some databases you can use automatically generated sequence numbers for keys with different options In the EMPLOYEES table for example you can generate even numbered sequences for one database and odd numbered sequence for the second database Doing so ensures that there are no conflicts between the databases 109 www it ebooks info CHAPTER 5 ADVANCED FEATURES Conflict resolution routines If you can t avoid key conflicts you need to identify that a conflict is about to occur and handle it appropriately based on business decisions and application logic You can add filters to your MAP statements to determine that a conflict is about to occur and how to handle it For example you can add a filter to determine if a record comes in with the same key and an older timestamp column than an existing record If so you can discard the older record For example on the HR EMPLOYEES table you can add a timestamp column to help you determine which row to discard on a SQL update statement conflict An example of the MAP statement to handle the conflict is as follows MAP HR EMPLOYEES TARGET HR EMPLOYEES amp REPERROR 90000 DISCARD amp SOLEXEC ID checkemployee ON UPDATE amp
243. e Collector is started dynamically for each Extract process on the source that requires a network connection to the target The Collector listens for connections on a port configured for GoldenGate Although it can be configured often the Collector process is ignored because it s started dynamically and does its job without requiring changes or intervention on the target During transmission from the source to the target you can compress the data to reduce bandwidth In addition you can tune the TCP IP socket buffer sizes and connection timeout parameters for the best performance across the network If needed you can also encrypt the GoldenGate data sent across the network from the source and automatically decrypt it on the target Collector The Collector process is started automatically by the Manager as needed by the Extract The Collector process runs in the background on the target system and writes records to the remote trail The records are sent across the TCP IP network connection from the Extract process on the source system either by a data pump or a Local Extract process Remote Trail The remote trailis similar to the source trail except it is created on the remote server which could be the target database server or some other middle tier server The source trails and the remote trails are stored www it ebooks info CHAPTER ARCHITECTURE in a filesystem directory named dirdat by default They are named with a two charact
244. e GoldenGate operations By default Goldengate stores temporary files in the dirtmp directory under the default installation directory A good rule of thumb to use for temp file space is around 10 GB of disk space e Plan on at least 10 MB per trail file As a rule of thumb we recommend that you start with at least 1 GB of disk space allocated per system for trail files Alternatively use the following formula that Oracle provides to determine the amount of disk space to set aside log volume in one hour x number of hours downtime x 0 4 trail disk space One way to calculate the total amount required for trail file space is by querying the V ARCHIVED_LOG view from within the source Oracle database The following query shows you how to do so select trunc COMPLETION TIME count 100 size in from v archived_log group by trunc COMPLETION TIME TRUNC COM SIZE IN MB 15 MAY 11 500 Run tests after installing Goldengate to measure your specific transaction mix and load and to gauge the total disk space required for trail files Network Requirements Since Oracle Goldengate software operates between source and target systems over networks you must configure TCP IP networking to accommodate all hosts within DNS to include host names that will be included in the Oracle Goldengate infrastructure deployed In the event of firewalls hosts must be allowed to send and receive data for open ports that the manager Extract and Repl
245. e GoldenGate is to open trail files for analysis and troubleshooting After starting a Logdump session you need to issue the OPEN command as shown here specifying the path name and trail file name Logdump 1 open dirdat aa00001 Current LogTrail is ggs orasrc dirdat aa00001 After you open the trail file you need to turn on some useful features to make full use of the Logdump utility The first command is GHDR ON It enables display of the record header for the transactions in the trail file for Oracle GoldenGate Logdump 2 ghdr on Next you need to toggle the option for viewing the hex and ASCII values for trailfile data To do so execute the detail data command Logdump 3 detail data Now you can move to the next record in the trail file by using the next command in Logdump When you issue the next command Logdump moves forward one record Logdump 5 next Hdr Ind 45 Partition x00 UndoFlag xoo BeforeAfter A x41 RecLength o xooo0 IO Time 2011 05 04 21 22 17 611 797 IOType 151 x97 OrigNode o TransInd x03 FormatType R x52 xoo Incomplete x00 299 www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR 300 AuditRBA 0 AuditPos 0 Continued N xoo RecCount o 2010 08 04 21 22 17 611 797 RestartOK Len 0 RBA 936 Name After Image PartitionO G s Here you can exa
246. e Online Redo Log section of this chapter to find out if seqno 187 is current or old If the current Oracle redo log seqno is 187 then there is no lag in the Extract If it s larger than 187 then the lag is on the source database side and the Extract can t keep up with the redo log generation This lag situation may be temporary for example during large updates But if the lag persists you need to tune the Extract Also note that the extract is writing to seqno 11 for the trail file c1 To confirm if the Data Pump is also reading the seqno 11 do the following GGSCI command GGSCI info dpggsi detail EXTRACT DPGGS1 Last Started 2011 04 12 18 29 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 04 ago Log Read Checkpoint File c ggs dirdat c1000011 2011 04 12 18 28 41 860000 RBA 994 Target Extract Trails Remote Trail Name Seqno RBA Max MB C ggs dirdat C2 8 1455 5 The data pump is also reading 1000011 seqno 11 so there is no lag If the data dump is reading a seqno less than 11 then the data pump can t keep up with the Extract This may be due to a network issue in which case you need to tune the Data Pump process Note that the data pump is writing seqno 8 on remote trail file c2 To check the replicat is in sync with the data dump run the following GGSCI command 161 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE GGSCI 22 info repggsi detail REPLICAT REPGGS1 Last Started 2011 04 12 18 28 Stat
247. e TCP socket buffer size is 30 000 bytes By increasing this buffer size you can send larger packets of data over the network The Oracle GoldenGate Windows and UNIX Reference Guide has a sample formula to help calculate the optimum TCP socket buffer size You should also check your specific server limitations for maximum TCP buffer size For example on Linux you can check the following settings to determine the maximum TCP transmit and receive buffer sizes net core rmem_max 262144 net core wmem_max 262144 In this example the transmit and receive buffers are set to 256KB You can check with your server and network administrator to determine if these sizes can be adjusted higher such as 4MB to allow GoldenGate to take advantage of higher TCP buffer sizes GoldenGate can utilize larger TCP buffer sizes if you add an option to the RMTHOST parameter as shown in the following example RmtHost targetserver MgrPort 7809 TCPBUFSIZE 100000 This example customizes the TCP buffer size to 100 000 bytes in the data pump Extract to try to improve performance by sending larger TCP packets Another network tuning option of the RMTHOST parameter is the COMPRESS option as shown in the following example RmtHost targetserver MgrPort 7809 TCPBUFSIZE 100000 COMPRESS Using COMPRESS GoldenGate compresses outgoing blocks of data over the network and then decompresses the data before writing it to the trail According to the GoldenGate Reference manual
248. e day to day management of your GoldenGate setup and ensure that it stays up and running smoothly Using GGSCI Command Shortcuts By now you should be familiar with the GoldenGate Software Command Interface GGSCI and many of the basic GGSCI commands Let s look at some shortcuts that may not be as familiar These can save you a lot of time and typing You can use common abbreviations in your commands such as ext for Extract rep for Replicat mgr for Manager and er to refer to both Extracts and Replicats Let s look at a few examples using the INFO command on Extracts Replicats and in the third example both Extracts and Replicats GGSCI server 1 gt info ext GGSCI server 2 gt info rep GGSCI server 3 gt info er Now let s use the exclamation point command to reexecute the last INFO command GGSCI server 4 gt info er Notice that the command reexecuted the info er command Next let s reexecute the second command info rep by adding the specific command number 2 to the as shown here GGSCI server 5 gt 2 info rep Suppose you want to execute an operating system command to display your current directory contents while you re inside GGSCI You can do this with the SHELL command as shown in the following example GGSCI server 6 shell dir You can quickly see information about your GoldenGate environment using the SHOW command SHOW is especially helpful to get a quick listing of y
249. e if you want to see the statistics reported permanently REPORTROLLOVER If an Extract or Replicat runs continuously for a long period you may notice that the report file becomes rather large To control the size of the report file you can have GoldenGate automatically roll over the reports on a daily or weekly basis The example rolls over the report file once a day at 1 15 a m Note that the report statistics continue to accumulate even after the reports are rolled over unless you specifically tell GoldenGate to reset them using the STATOPTIONS parameter with the RESETREPORTSTATS option Reporting Discarded Records In some cases you may notice that GoldenGate is unable to process particular records for one reason or another These records are called discards by GoldenGate For example you may have changed a table layout and forgotten to update your GoldenGate configuration In that case GoldenGate may discard the records for that table because it doesn t know about the new table layout By default GoldenGate won t produce a discard file to show you which records have been discarded If you want to see the discards you can instruct GoldenGate to write them out to a file using the DISCARDFILE parameter The following example tells GoldenGate to generate a discard file for the human resources schema Extract named LHREMDI You also have any new records appended to the discard file with the APPEND parameter To keep the discard file from ge
250. e issues Data Synchronization Issues As much as you might like Oracle GoldenGate to automate the data verification process it doesn t perform these checks However you can implement conflict resolution scripts and programs to check for data synchronization between source and target systems Oracle GoldenGate performs basic checks by matching target rows from the source and target database primary key and unique key columns or substitute key columns if you define these with the KEYCOLS option for the TABLE and MAP statements in the Extract and Replicat You should perform out of sync data checks on a regular basis as part of your database administration duties for Oracle GoldenGate to remediate synchronization issues Veridata provides a suite of tools to check for data synchronization issues with Oracle GoldenGate environments www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE What causes data synchronization issues you may ask There are several key factors e Tables missing key constraints e Character set configuration issues e Column missing errors e Fetch failures on source and target tables Tables Missing Key Constraints Oracle GoldenGate relies on table key constraints to map source to target DDL and DML activities Without these keys Oracle GoldenGate uses all of the table columns as a key value This may cause multiple rows of the table to be updated if they contain identical data One way to address this i
251. e parameters for bidirectional replication you should be aware of the following considerations e You can t replicate truncates in bidirectional replication You can work around this restriction by configuring truncates for one way replication as long as they always originate from the same server e From a practical standpoint bidirectional replication is typically done with a maximum of two identical databases Any more than that and the configurations become extremely hard to manage e Inan active active configuration it isn t possible to disable triggers so you should add code to your triggers to ignore DML generated by the GoldenGate Replicat e Ifyou re using cascade delete constraints these need to be disabled and converted to triggers 107 www it ebooks info CHAPTER 5 ADVANCED FEATURES 108 e You need to plan for database generated sequential keys so there is no overlap across databases For example you can have one database generate odd key values and the other database generates even key values Another option is to have one database use a specific range of keys and the other database use a different range of keys to avoid duplication e Ifyou share the same key values across both databases then you must develop some conflict resolution routines For example if the same data on a row with the same key is being updated on both databases which update should be applied GoldenGate provides documentation example
252. e replicated between like platforms such as from Oracle 10g to Oracle 11g environments However we will show you how to set up heterogeneous environments in the easiest manner possible You will learn how to prepare the environment from source to target database configurations as well as administration tasks involved with heterogeneous replication with Oracle GoldenGate environments Since heterogeneous replication can involve a large variety of third party database platforms the amount of time and space to address each configuration would fill a volume in and of itself As such we have chosen to present a case study example of common platforms that we have implemented for customers in the field Our case study will provide information on how to configure and implement heterogeneous replication for Oracle to Microsoft SQL Server The first task in setting up heterogeneous replication for Oracle GoldenGate involves consultation of the Oracle GoldenGate documentation with a focus on review of the Oracle GoldenGate release notes to find up to date changes in heterogeneous replication with Oracle GoldenGate Microsoft SQL Server to Oracle Replication Microsoft SQL Server is one of the most popular and widely known RDBMS platforms for small to mid size corporate database environments for back office operations However it does not scale well based on our experiences for large enterprise data warehouse environments that migrate to massive terabyte and eve
253. e sql script to enable the DDL trigger SOL ddl_enable sql Trigger altered 7 Update the local Extract parameter file to include the DDL parameter to tell GoldenGate you wish to replicate DDL In this example you re only replicating DDL statements that are mapped in your replication scenario You can choose to INCLUDE other tables or EXCLUDE them from DDL replication using different parameter options as well DDL Include Mapped At this point you should be replicating DDL changes between your Oracle databases Next you see how to set up bidirectional replication Adding Bidirectional Replication In Chapter 4 you set up basic one way replication Let s expand that basic configuration now to support bidirectional replication First let s review from Chapter 3 the architecture for bidirectional replication In bidirectional replication as shown in Figure 5 2 changes to data can occur on either database at the same time and are replicated to the other database Each database contains an identical set of data Bidirectional replication is sometimes called active active replication because each side of the replication is actively processing data changes Source Target Source Target Database Database Figure 5 2 Bidirectional replication Figure 5 3 illustrates a typical bidirectional GoldenGate configuration Such a configuration is often used in environments where there are high volume and high performance requirements By
254. ed databases www it ebooks info CHAPTER ARCHITECTURE Table 3 1 GoldenGate 11g Source Databases Source Database c tree DB2 for Linux UNIX Windows DB2 for z OS MySQL Oracle SQL MX SQL Server Sybase Teradata Capture Local Extract Process Capture is the process of extracting data that is inserted into updated on or deleted from the source database In GoldenGate the capture process is called the Extract In this case the Extract is called a Local Extract sometimes called the Primary Extract because it captures data changes from the local source database There are several types of extracts Another type of Extract that s discussed later is the data pump Extract which passes the Local Extract changes to the target server You can also have an initial load Extract to capture database records to perform an initial load of the target database You see an example of the initial load Extract in Chapter 4 Basic Replication Extract is an operating system process that runs on the source server and captures changes from the database transaction logs For example in an Oracle database Extract captures changes from the redo logs and in some exceptional cases the archived redo logs and writes the data to a file called the Trail File For Microsoft SQL Server Extract captures changes from the transaction log To reduce the amount of processing Extract only captures committed changes and filters out
255. ed 00 00 03 File dirdat 12000013 2011 03 25 23 21 48 001492 08 Status RUNNING ago RBA 2852 111 Status RUNNING ago RBA 2852 11 Status RUNNING ago RBA 2852 BATCHSOL is another parameter you can add to your Replicats to improve performance According to the Oracle GoldenGate documentation BATCHSQL can improve the performance of Replicats by as much as 300 percent There are some usage restrictions for BATCHSOL and it s only valid for Oracle SQL Server DB2 LUW DB2 on z OS and Teradata databases You should check the Oracle GoldenGate Windows and UNIX Reference Guide for the full list of restrictions 146 www it ebooks info CHAPTER 7 88 TUNING In normal processing GoldenGate groups transactions together but preserves transaction order from the source With BATCHSOL GoldenGate groups together similar transactions into a high performance array and processes them at a much faster rate To implement BATCHSQL using defaults simply add the following parameter to your Replicat parameter file BATCHSQL You can add several optional parameters to BATCHSQL if you need to tune the amount of memory BATCHSQL is consuming You can refer to the GoldenGate Reference Guide for details on those parameters Typically you should start with the default value and then add the optional parameters if needed Let s look at a few examples to illustrate how BATCHSOL works Figure 7 5 shows GoldenGate processing a remote trail normally
256. ed in the example You can refer to the Oracle GoldenGate Reference Guide for a detailed description of all the parameters I usually refer to the Alphabetical Reference section in the Oracle GoldenGate Reference Guide to quickly find a parameter and its meaning and various options Tip You can use two dashes at the beginning of a line in the parameter file for comments Let s start by reviewing the Replicat parameter in the next section Replicat As with the initial load Replicat use the Replicat parameter to define the GoldenGate ongoing change Replicat group You should follow naming standards as covered earlier In the example the Replicat group is named RHREMDI The R specifies that it s the ongoing change Replicat HREM specifies that you re using this Replicat group for the human resources employee application and the HR schema The D signifies that you re running in the development environment Any finally the 7 indicates this is the first Replicat group for this schema and application You can add additional parallel Replicat groups to improve performance as covered in Chapter 7 SETENV As we showed in the initial load Replicat you can use the SETENV parameter to override default environment variable settings In the example you explicitly set NLS_LANG to AMERICAN AMERICA AL32UTF8 It s a good idea to set NLS LANG and not let it default to the environment setting It is possible we could have a different NLS LA
257. ed in this chapter are a foundation to build on for those more advanced configurations Chapter 5 also builds on this basic replication configuration and adds some advanced features that you can use to enhance the basic configuration Prerequisites for Setting Up Replication You need to have the following prerequisites in place before you can start setting up replication e The GoldenGate software installed on the source and target servers as described in Chapter 2 e The GoldenGate database user ID created on the source and target databases e server name or IP address of the target database server e The GoldenGate Manager process up and running on the source and the target 49 www it ebooks info CHAPTER 4 M BASIC REPLICATION e TCP IP network connectivity open from the source server to the target server s GoldenGate manager port e Anunderstanding of the business and technical replication requirements Let s dig a little deeper on the last item in the next section about requirements Understanding the requirements is probably the single most important factor in the success of a replication project Requirements You need a solid understanding of the specific requirements that drive the replication technical design Here are some of the factors that impact the replication design Source and target configurations Describe the existing or planned source and target databases What are the software releases for the database a
258. ed on the client machine e Client host must be on the same physical network as that for the Oracle GoldenGate Director Server GoldenGate Director Web Client Installation Guidelines The Oracle GoldenGate Director web client machine must contain a valid and supported Internet web browser The following browsers are supported e Mozilla Firefox 1 4 or later e Microsoft Internet Explorer 5 0 or later e Apple Safari 1 2 or later 26 www it ebooks info CHAPTER 2 88 INSTALLATION Installing Oracle GoldenGate Director Server Once you have satisfied all of the above prerequisites the installation for the Oracle GoldenGate Director Server can be performed First you will need to create a new database schema account in the repository database on the Oracle GoldenGate Director Server Then you can perform the installation of the Oracle GoldenGate Director software Grant Database Privileges and Credentials to Oracle GoldenGate Director Server Schema If you choose to use Oracle for the repository database you will need to create a schema user account and grant the QUOTA UNLIMITED privilege to the user s default tablespace For details on using other database platforms such as MySQL and Microsoft SQL Server as the Oracle GoldenGate Director repository database please consult the Oracle GoldenGate Director Administrator s Guide 11g Release 1 11 1 1 documentation available online at http download oracle com docs cd E18101 01 doc 1
259. edia Pack Search EN E Delivery Search Download Media Pack Search 5 Instructions Frequently Asked Questions Review the License List to determine which Product Pack or What is a Medis Pack Packs you need to download If 2 Select the Product Pack and Platform and click Go 3 If there is only one result you will see the download page If How do I get my license there are multiple results select one and click Continue code More Select a Product Pack Oracle Fusion Middleware 2 Platform icrosoft Windows 32 bit ad EJ Figure 2 1 E Delivery website for obtaining Oracle Goldengate software Once you have chosen the correct platform you will see choices for the Oracle GoldenGate software to download as shown in Figure 2 2 www it ebooks info CHAPTER 2 88 INSTALLATION pem on ma em Orace Fusion Middleware 11g Media 11 1 1 3 0 B55383 24 DEC 17 2010 102 Pack Microsoft Windows 32 bit 57G Oracle GoldenGate Application Adapters 11 1 1 0 0 B59968 01 AUG 12 2010 16 11 1 1 0 0 for JMS and Flat File Media 433M Pack Orade GoldenGate for Non Oracle 11 1 1 0 0 B60580 02 NOV 04 2010 6 Database v11 1 1 0 0 Media Pack for 62M Microsoft Windows 32 bit Management Pack for Oracle 11 1 1 0 0 B60816 01 OCT 21 2010 4 GoldenGate 11 1 1 0 0 Media Pack 583M Oracde GoldenGate for Non Oracle 10 4 0 0 0 B57525 04 JUL 15 2010 5 Database 10 4 0 Medi
260. eger STARTTIME date time string ENDTIME date time string SYSKEY lt gt 32 64 bit syskey SYSKEYLEN comparison value TRANSIND lt comparison gt nn UNDOFLAG comparison nn RECLEN comparison nn AUDITRBA comparison nnnnnnnn ANSINAME ansi table name GGSTOKEN tokenname lt comparison gt tokenvalue USERTOKEN tokenname comparison tokenvalue CSN LogCSN lt comparison gt lt value gt lt column range gt lt start column gt lt end column gt ie 0 231 lt comparison gt lt gt lt gt lt gt EQ GT LE GE LE NE X lt program gt string Execute lt program gt TRANSHIST nnnn Set size of transaction history TRANSRECLIMIT nnnn Set low record count threshold TRANSBYTELIMIT nnnn Set low byte count threshold LOG STOP TO filename Write a session log BEGIN date time Set next read position using a timestamp SAVEFILECOMMENT on OFF Toggle comment records in a savefile SAVE lt savefilename gt options Write data to a savefile options are nnn RECORDS nnn BYTES NOCOMMENT Suppress the Comment header trailer recs Default COMMENT Insert Comment header trailer recs OLDFORMAT Force oldformat records NEWFORMAT Force newformat records TRUNCATE purgedata an existing savefile EXT pri sec lt max gt Savefile Extent sizes
261. em to the source trail file 2 Start the data pump Configure and start the data pump to send extracted changes captured by the local Extract into the source trail across the network to the target server When on the target server the changes are written to the remote trail file 3 Perform the initial load Load the data using either GoldenGate or vendor DBMS utilities Doing so creates a synchronized copy of each database except for changes that were captured during the load process by the Extract These changes will be applied in the next step when the Replicat is started You only need to run an initial data load the first time you setting up replication After this GoldenGate will keep the data synchronized on an ongoing basis using the Extract and Replicat processes 4 Start the Replicat Configure and start the Replicat to begin applying changes from the remote trail file Changes that were captured while the initial load process was executing have already been written to the remote trail file It may take a Replicat some time to catch up with the changes depending on the data volume and the length of time changes were captured during the load process Also some of these changes may cause data conflicts because activity was occurring in the database at the same time the data load was running Later this chapter covers some GoldenGate parameters such as HANDLECOLLISIONS to automatically handle these data conflicts When the Replicat
262. ence number We can get the maxiumum sequence for c1 and c2 by listing the directory files 1s 1 Linux Unix or dir in Windows The maximum is the current trail file for the process For examples Extract is writing the C1000011 file but if the Data Pump process is reading anything other than C1000011 file then we have lag in the Data Pump process C1000011 is connected between Extract and Data Pump processes Whereas C200008 is connected between Data Pump and Replicat processes 160 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE Check the maximum Sequence for each trail file as shown here GGSCI STRICHCHUT61 21 shell dir dirdat Directory of C ggs dirdat 04 12 2011 06 28 PM 994 C1000010 04 12 2011 06 28 PM 994 C1000011 04 12 2011 06 29 PM 1 411 C2000007 04 12 2011 06 29 PM 1 455 C2000008 The maximum Sequencett for c1 is 11 and for C2 is 8 as shown above To confirm if the extract is writing to the current trail file without any lag we can use the following GGSCI command GGSCI gt info extggsi detail EXTRACT EXTGGS1 Last Started 2011 04 12 18 28 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 05 ago Log Read Checkpoint Oracle Redo Logs 2011 04 12 19 13 40 Seqno 187 RBA 33210880 Target Extract Trails Remote Trail Name Seqno RBA Max MB C ggs dirdat C1 11 994 5 The Extract is reading the redo log seqno 187 You can use the query in the Checking the Current Oracl
263. enever the DISCARDFILE parameter is used by either the Extract or Replicat If there is a problem during processing of data from the source or target environments by either the Replicat or Extract then the rejected records are dumped into the discard file You should review the contents of the discard file on both the source and 223 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 224 target database environments on a regular basis as part of your due diligence to catch data issues encountered by Oracle GoldenGate The discard file contains all column level details for database operations that Oracle GoldenGate processes couldn t process successfully Each discard file contains the following useful details e Database error message for the process such as ORA 00100 for No Data Found with Oracle 11g e Thetrail file sequence number for record s that the Extract or Replicat attempted to be processed e The relative byte address of the record in the trail file e Details of the discarded record that the Extract or Replicat attempted to process Typically a discard file is used by the Replicat to log operations that couldn t be reconstructed or applied but you may find one useful for the Extract as well Check the discard error for messages such as ORA 1403 or duplicate and or rejected records Here is a sample Oracle GoldenGate discard file ORA 20017 724935 ORA 06512 at HR REPORAINSERT line 4
264. eplicat disaster recovery replication 247 248 starting 276 278 stopping 275 277 REPLICAT keyword 59 Replicat MAP parameter 230 Replicat object altering RUN options for 206 321 www it ebooks info INDEX REPLICAT parameter 72 79 231 Replicat parameters 270 273 286 configuring for Replicat process 78 files 72 74 225 288 Replicat processes 39 76 83 Abend failures 236 237 adding 81 configuring 77 81 ASSUMETARGETDEFS parameter 80 HANDLECOLLISIONS parameter 79 80 MAP parameter 80 81 Replicat parameter 78 SETENV parameter 78 USERID parameter 79 for zero downtime migration 270 271 editing Run Options 206 errors 236 failing on large transactions 238 fallback configuring for zero downtime migration 273 274 GoldenGate 191 incompatible record errors 238 parallel implementing with table filtering 135 141 overview 134 135 322 using key ranges 141 146 reporting by 86 89 REPORT parameter 88 89 REPORTCOUNT parameter 88 REPORTROLLOVER parameter 89 stalling 236 starting and stopping 81 verifying 82 83 replication 49 83 85 110 advanced 2 3 basic 2 bidirectional 42 43 106 110 excluding transactions for 108 109 handling conflict resolution for 109 110 broadcast 44 data real time 182 data filtering columns 95 96 rows 96 tables 94 95 data mapping columns 97 98 data pump 65 70 adding 67 configuring 65 67 starting and stopp
265. eplicat is added it establishes checkpoints in the trail file to keep track of processing The EXTTRAIL parameter of the ADD REPLICAT command tells GoldenGate to use a specific trail file for the Replicat to process The example uses trail file 12 as the source for the Replicat This is the trail file that was written by your data pump Extract In the example you take the defaults and start processing at the beginning of the first trail file You can also tell Replicat to begin applying changes at a specific timestamp or using a specific trail file number Tip You should store commands such as ADD REPLICAT in an obey file in the diroby directory You can execute these from the GGSCI command prompt using the obey filename command This is good practice because the commands are readily available as a future reference or if you need to repeat the commands at some point Chapter 14 covers obey files in more detail Now you can proceed with starting the Replicat Starting and Stopping the Replicat After adding the Extract you need to start it to actually begin applying changes to the target database The following example starts the Replicat RHREMDI GGSCI targetserver gt START REPLICAT RHREMD1 If needed you can stop the Replicat using a similar method For example you may need to make a change to the GoldenGate Replicat parameters To do that you stop the Replicat make the changes to the parameter file and then start the Replicat for
266. er Oracle Teradata has a number of requirements that must be performed prior to installing Oracle Goldengate Before Oracle Goldengate can be installed for Teradata the following items must be configured and available e Change Data Capture CDC e Replication Groups e Teradata Access Module TAM e Relay Services Gateway RSG The setup and configuration of these tasks for Teradata is beyond the scope of this chapter For details on how to configure these features with Teradata please consult the Teradata documentation available online at ww info teradata com System Requirements for Teradata with Oracle GoldenGate Oracle GoldenGate has unique requirements for setup with Teradata The Oracle GoldenGate replication server must be configured along with setups for the source and target Teradata systems if used Oracle GoldenGate Replication Server Itis important to keep the following points in mind when deploying Oracle GoldenGate for Teradata e Install Oracle 11g GoldenGate on a separate physical server Do not install on any of the Teradata Servers e Teradata Access Module TAM must be installed on the Oracle GoldenGate replication server under the root directory for the Oracle GoldenGate installation directory TAM communicates with an Oracle GoldenGate API called the Vendor Access Module VAM Details for TAM configuration are provided in the Teradata Replication Services documentation available online at www info te
267. er prefix followed by a six digit sequence number The same approach for sizing for the source trail applies to the remote trail You should size the trail files based on the expected data volume When the specified size is reached a new trail file will be created You can also configure GoldenGate to automatically purge the remote trail files based on age or the total number of trail files to free up disk space Just like the source trail data in the remote trail files is stored in platform independent GoldenGate proprietary format Each remote trail file contains a file header and each record also contains its own header The GoldenGate processes keep track of its position in the remote trail files using checkpoints which are stored in separate GoldenGate files or optionally in a database table You can also examine the remote trail files in detail just like the source trails by using the GoldenGate Logdump utility You learn more about the Logdump utility in Chapter 11 One thing to keep in mind is you should never manually edit and change the trail files using a text editor Delivery Replicat Delivery is the process of applying data changes to the target database In GoldenGate delivery is done by a process called the Replicat using the native database SQL The Replicat applies data changes written to the trail file by the Extract process in the same order in which they were committed on the source database This is done to maintain data in
268. ermediate and target systems If the remote trail file on the target system fails to empty on a regular basis check to see if your Replicat process groups are available running and processing data from the remote trail files If the Replicat processes are running correctly without errors the cause may be large transactions being processed by the Replicat on the target system In this case you can split up the tables and partition them to break out the large transactions into multiple trail files with additional Replicat process groups to process the transactions in parallel To do so use the Oracle GoldenGate Replicat parameter FILTER option with the QRANGE function in the MAP statement for the Replicat Trail Files that Don t Roll Over Another issue that plagues Oracle GoldenGate trail file operations occurs when a trail file fails to roll over to a new trail file after it fills up completely One thing to check here is the maximum file size for the trail file in question You can check this using Oracle GoldenGate GGSCI command INFO EXTTRAIL or INFO RMTTRAIL as shown in the following example GGSCI oracledba 1 info exttrail Extract Trail AA Extract EXTORA Seqno 0 RBA 0 File Size 10M You can locate this information by examining the File Size field To change the trail file size with Oracle GoldenGate issue the Oracle GoldenGate GGSCI command ALTER EXTTRAIL or ALTER RMTTRAIL with the MEGABYTES option as shown here GGSC
269. ers can continue normal processing using the standby database Any database DML changes made to the standby database will be accumulating in the GoldenGate source trail files on the standby server until the primary database is repaired or replaced and brought back online Once the primary database server is repaired or replaced and back online you can begin part 3 of the process of moving the applications and users back to the primary database The first steps of this process are shown in Figure 12 11 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION Restore GG Files Start the Manager 1 3 E ue T Delete Add the Extract 2 Primary D Local Database Extract Source Trail Figure 12 11 Disaster recovery unplanned failover part 3 Following are descriptions of the part 3 steps in Figure 12 11 1 Restore the GoldenGate software directories and files from your offline server backups 2 Delete the Local Extract and the source trails and then add the Local Extract back as shown in the following example Do not start the Local Extract until later GGSCI primaryserver DELETE EXTRACT LHREMD2 GGSCI primaryserver gt DELETE EXTTRAIL dirdat 13 GGSCI primaryserver ADD EXTRACT LHREMP2 TRANLOG BEGIN NOW GGSCI primaryserver ADD EXTTRAIL dirdat 13 EXTRACT LHREMP2 MEGABYTES 100 Deleting the Local Extract and trails reinitializes them so you can start over with extracting changes The delete extt
270. es on the target server as covered in the section Prerequisites for Setting Up Replication When you ve confirmed that the prerequisites are met you can begin configuring the Replicat In order to do this you first need to create a parameter file for the Replicat Remember you re configuring the Replicat to apply all the DML changes from the HR schema Note You can edit the GoldenGate parameters directly from the GGSCI or using the native OS text editor outside of GGSCI whichever is most convenient To edit in GGSCI you enter EDIT PARAMS paramterfilename GGSCI brings up the parameter file in the default text editor which is Notepad for Windows and vi for UNIX and Linux You can change the default editor using the SET EDITOR command in GGSCI If you re editing directly from the OS save the parameter file in the dirprm subdirectory under the GoldenGate software installation directory In the example this is gger ggs dirprm Let s begin by taking a look at the Replicat parameters for RHREMDI as shown in the following example GGSCI targetserver 1 edit params RHREMD1 Replicat RHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 77 www it ebooks info CHAPTER 4 M BASIC REPLICATION 78 USERID GGER TargetDB PASSWORD userpw HandleCollisions AssumeTargetDefs Map HR Target HR Let s examine the parameters in more detail Keep in mind that GoldenGate has many more configuration parameters than are us
271. es usage of various default ports via TCP IP networking The following ports are used by default and should be made available e 4150 Server communication port e 8820 shutdown port e 8830 HTTP port Note These default ports for Oracle GoldenGate Veridata can be changed after the installation In chapter 9 we will perform an installation of the Oracle GoldenGate Veridata components Additional information on how to install Oracle GoldenGate Veridata is also available from the Oracle GoldenGate Veridata Administrator s Guide available online at http download oracle com docs cd E15881 01 doc 104 gg veridata admin pdf Install Oracle Goldengate Veridata The following steps are required to get started with the Oracle GoldenGate Veridata installation 1 Download the Oracle 11g Goldengate Veridata software 2 Unzip the Veridata software to a directory that you will use to stage the software Summary In this chapter we provided you with a detailed roadmap to install the Oracle GoldenGate product family of data replication applications Common installation issues such as infrastructure setup for network operating systems and storage were provided to help you understand common pitfalls frequently encountered in the deployment process for Oracle GoldenGate 31 www it ebooks info CHAPTER 3 Architecture This chapter looks at the typical GoldenGate replication flow and examines each of the architectural components in
272. eseeass 63 Verifing the Supe aai in 63 Starting the Data 65 Configuring the Data 65 Adding the Data 67 Starting and Stopping the Data Pump cccsssssssssessssssssssesessssessssecsesecsessesessesesaesesaesaesesassesaeseseasanaes 68 Verifying RUOTE 68 70 Prerequisites for the GoldenGate Initial 0 70 Configuring the Initial Load Extract cccsscsssscsssessesssessssecssssesessssessssessesesaesessesassesaseesassesaesassesaseees 71 Adding the Initial Load 1 71 Configuring the Initial Load 72 Adding the Initial Load RepliCat eene ntn ntn ntn tenes 74 Starting the GoldenGate Initial Load essent netten tentent tnntntnns 74 Verifying the Initial Load eeeeeeeeeere rennen nne nnne nnn nennt nnn nnn natnra tn 74 Loading with DBMS UUTOS 75 Prerequisites for Loading with DBMS Utilities eeeeeeneneenneneettntnnnt
273. ess before doing this in production or the data will be lost if you skip the records inadvertently Changing the Trail File Size If Oracle GoldenGate generates too many trail files it s difficult to maintain them A trail file that s too big is also risky if a large trail file is corrupted then you lose alot of data You need to monitor and adjust the trail file size if the source activity changes and affects the trail file generation rate To change the trail file size click the Trail object in the diagram and choose Controls Run Options Then you can enter the new trail file size Extracting Tranlogoptions By default Director doesn t include tranlogoptions The tranlogoptions parameter controls the way Oracle GoldenGate interacts with the transaction logs it s database specific To add tranlogoptions www it ebooks info 10 GOLDENGATE DIRECTOR select it in the list of parameters shown in Figure 10 22 and then click Insert To get to the following dialog click on the Insert button on the Parameter tab on any mapping icons Insert Parameters x 4 Value sourceisfile ftranlogoptions sourceistable specialrun r Attributes sglexec statoptions E targetdefs Path Map tcpsourcetimer threadoptions Alt Archive Log Format titrace trace Archived Log Only Jarchivediogonly 3j trace2 tracetable Max Warn EOF 0 8 E ee af
274. esses will show that the primary Extract s checkpoints are moving because it s able to write to the local trail however the data pump checkpoints aren t incremented thus showing the network latency issue When you encounter this latency issue contact and work with your local system administrator or network operations team to remediate the network slowness Further analysis can be performed by issue the INFO EXTRACT and INFO REPLICAT commands from within the GGSCI interface with the DETAIL option INFO EXTRACT lt group gt DETAIL INFO REPLICAT lt group gt DETAIL 233 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 234 Given that the Replicat process groups are up and running without any system issues in terms of access to reading from and writing to the trail files the next step is to compare the Extract write point with the Replicat read point by reviewing the following statistics output from the GGSCI INFO EXTRACT command Remote Trail Name Seqno RBA Max MB c ggs_src dirdat aa 1 251 50 Compare this to the report from the GGSCI command INFO REPLICAT Log Read Checkpoint File C ggs trgt DIRDAT AA000001 2011 03 13 22 41 58 000000 RBA 2142225 Whenever you see that the Extract checkpoints aren t being incremented INFO REPLICAT may show continued processing by indicating that the read relative byte address RBA is increasing up until the last record has been written to the trail file It then stops
275. essing rate is 154 and the delta is 215 The delta is higher which indicates that you re processing more records in the latest reporting period You can compare the rate over time to see if it changes Changes in rate could be due to more or less processing volume or perhaps performance issues with your Extract or Replicat or even the database itself REPORT You can use the REPORT parameter to have the Extract or Replicat insert runtime statistics into your report such as the following FRR RAR KK RK koe EK EK EK EK SKK KKK KKK KK KKK KKK Run Time Statistics 2K OK OK SE KK KK OK E SE SK SK SK SK SK SK SK SK SK 2 2K SK SK OK OK OK KK SE E OK OK OK OK GE RE SE SE SE SE SK SE SE OK SK OK OK Report at 2011 01 15 13 53 30 activity since 2011 01 13 23 23 10 Output to gger ggs dirdat l2 88 www it ebooks info CHAPTER 5 ADVANCED FEATURES From Table HR EMPLOYEES inserts 6 updates 2 deletes 1 discards 0 The example told the Extract or Replicat to generate statistics every day at 1 a m You can also picka specific day of the week and generate statistics only once a week Tip You can also manually add runtime statistics to your report using the SEND EXTRACT xxx REPORT or SEND REPLICAT xxx REPORT where xxx is your Extract or Replicat name Remember to update your parameter fil
276. et 4 23 11 1 46 PM 4 23 11 2 32 PM 4 23 11 7 01 PM Job Run 4 23 11 7 00 PM Figure 9 7 Performance statistics You also can view detail performance in the Compare Pair Reports in the same screen by clicking the Report link The following are the sample Compare Pair Reports Processing first rowhash block from source at 2011 04 23 19 01 09 Processing first rowhash block from target at 2011 04 23 19 01 10 Performance Statistics for source Rowhash at 2011 04 23 19 01 10 duration secs row bytes sec 73236 00 00 01 73236 00 7937097 7937097 bytes row 108 IOWS rows sec row bytes rh bytes row 4 rows skipped 333760 www it ebooks info CHAPTER 9 E ORACLE GOLDENGATE VERIDATA blocks skipped hash comp rate total comp rate pct time fetching pct time waiting time until first row 00 00 00 ipc msgs 12 ipc bytes 2337393 bytes msg 194782 compressed bytes msg 72710 bytes sec 2337393 compressed bytes sec 872521 msg compression ratio 0 37 SONS NSORT statistics Nsort version 3 4 29 Windows x86 32 bit using 49M of memory out of 50M Pointer sort performed Sat Apr 23 19 01 08 2011 Input Phase Output Phase Overall Elapsed 2 03 0 08 2 11 I 0 Busy 2 02 100 0 01 100 2 03 Action User Sys Busy User Sys Busy User Sys Busy main 0 00 0 00 0 0 00 0 00 0 0 00 0 00 0 1 0 14 0 00 7 0 00 0 00 0 0 14 0 00 7 2 0 12 0 00 6 0 02 0 00 25 0 14 0 00 7 All 0 26 0
277. et to the Tomcat Web Server Administration Tool screen as shown in Figure 9 8 http lt hostname gt lt port gt admin The default Apache Tomcat Web Server port is 8830 To change the default port modify the server xml file in the Veridata install dir server web conf directory Search for 8830 and replace it with another port number 186 www it ebooks info CHAPTER 9 88 ORACLE GOLDENGATE VERIDATA Crores ADMINISTRATION TOOL Tomcat Server Roles 6 1 Senice Catalina List Resources amp Data Sources Mail Sessions Role Name Description 1 Environment Entries Gk User Databases 9 CUser Definition Users Roles Figure 9 8 User definition roles in Tomcat Web Server Administration Tool Here are the descriptions of the role names e Administrator This is the admin id that you use for the login initially This role can perform all functions e DetailReportView This is the ReportViewer role but it can also view the out of sync report information through the Web interface or at the file level Some business sensitive information may be in the detail reports e PowerUser As with the admin role you can do everything except any configuration function on the server side e ReportViewer With this role you only can view the reports jobs and configurations 187 www it ebooks info CHAPTER 9 ORACLE GO
278. ew them for potential performance problems You can examine the lag and determine if it s too high to meet the business requirements For example suppose that during testing you experience a peak lag of 15 minutes on a critical Replicat Your business requirement says that any lag over ten minutes for this Replicat is a serious problem In this case additional tuning hardware or design changes are required to bring down the lag 287 www it ebooks info CHAPTER 14 TIPS AND TRICKS 288 Limiting the Number of Extracts You should try to extract as much data as possible with a single Local Extract The Extract processes usually aren t a performance bottleneck because they re reading the data Performance lag usually occurs in the Replicats that are processing changes and not the Extracts In addition if the data needs to be filtered use the data pump to create separate streams to send to the Replicat For example let s assume you re extracting data from the CUSTOMER table and replicating to two different databases one for reporting and the second for high availability The CUSTOMER table should be extracted from the source database only once and you can use two data pumps to filter and send the data to each of the different target databases Using Passthru Mode for Data Pumps If the data pump is only passing the data on to the target system and not doing any filtering use the data pump Extract PASSTHRU parameter This causes the
279. ext you will add the standby Extract to the GoldenGate configuration so you will be ready to start it later during a database switchover or failover You can do that using the following commands from GGSCI GGSCI standbyserver ADD EXTRACT LHREMP3 TRANLOG BEGIN NOW GGSCI standbyserver gt ADD EXTTRAIL dirdat 15 EXTRACT LHREMP3 MEGABYTES 100 Next let s configure the standby data pump Configuring the Standby Data Pump Now that the standby Local Extract is added you can proceed with configuring and adding the standby data pump The standby data pump is similar to the primary data pump except it will be used to pump the trails back from the standby database to the primary database if needed In our example you ll configure the data pump to read the 15 trail file written out by the standby Local Extract named LHREMP3 and pump it over the TCP IP network to the target server to be processed by the standby Replicat From GGSCI let s edit the parameters for our data pump as shown in this example 249 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION GGSCI standbyserver 1 gt edit params PHREMP3 Extract PHREMP3 PassThru ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt PHREMP3 dsc Append DiscardRollover at 02 00 RmtHost standbyserver MgrPort 7809 RmtTrail dirdat 16 Table HR Since you ve reviewed these parameters previously in Chapters 4 and 5 let s proceed with adding
280. file is to use the CHECKPARAMS parameter to verify syntax Like a spell checker CHECKPARAMS verifies the syntax whenever the Oracle GoldenGate process starts It writes results to the report file and then stops the process Note After using the CHECKPARAMS parameter and resolving syntax errors remove the parameter from the process group parameter file Otherwise the process won t restart Some common syntax errors that occur in parameter files for Extracts and Replicats include the following e Extract TABLE parameter or Replicat MAP parameter isn t terminated with a semicolon e Commas aren t followed by a space in parameter files e Missing commas quotes or parentheses in nested clauses exist such as a COLMAP Operating System Configuration Issues with Oracle GoldenGate Oracle GoldenGate relies on the operating system platform to perform replication activities on a real time basis All the core processes for Oracle GoldenGate operate as system background processes either as Linux Windows daemon processes or as Windows services for Windows If there is a hiccup at the operating system level Oracle GoldenGate processes fail Let s take a closer look at some common operating system issues that cause failures in Oracle GoldenGate and how you can resolve them First are configurations for the operating system that are missing system libraries In the case of UNIX and Linux operating systems if you see errors in Oracle Go
281. file for the Extract or Replicat Here s an example C ggs_src gt extract paramfile C ggs_ src dirprm extora prm Version 6 0 Build 6002 Service Pack 2 Process id 6056 Description 2011 03 14 00 19 46 ERROR 0066 00664 OCI Error beginning session status 10 17 ORA 01017 invalid username password logon denied 2011 03 14 00 19 46 ERROR OGG 01668 PROCESS ABENDING On UNIX and Linux platforms you need to look for a core dump file located under the Oracle GoldenGate home directory where you ve installed the Oracle GoldenGate software Oracle GoldenGate Trail File Problems Recall from earlier chapters that Oracle GoldenGate process groups write and read data to special flat files called trail files on the source and target systems Failure in processing these trail files halts correct operations of the Oracle GoldenGate software This section looks at the following problems that can occur with trail files e Trail files that don t empty e Trailfiles that don t roll over e Trail file purging issues Trail Files that Don t Empty Sometimes you encounter a problem with trail files that don t empty and remain full of data You should check to identify whether the trail file in question is a local or remote trail file If the problem is with a local trail file that fails to empty check for TCP IP network latency issues between the source 221 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 222 int
282. fo CHAPTER 6 HETEROGENEOUS REPLICATION GGSCI oracledba 5 add rmttrail dirdat lt extract extora megabytes 5 RMTTRAIL added GGSCI oracledba 6 gt info rmttrail Extract Trail dirdat lt Extract EXTORA Seqno 0 RBA 0 File Size 5M Create the extract parameter as shown in the following sample parameter file Change Capture parameter file to capture EMP and DEPT Changes EXTRACT EXTORA USERID GGS PASSWORD GGS RMTHOST localhost MGRPORT 2000 RMTTRAIL dirdat lt TABLE GGS EMP TABLE GGS DEPT Start the capture process for Oracle on the source system by using the START command in GGSCI as shown For the change capture delivery process configure the Replicat parameter and add the trail files in the normal manner from previous chapter examples SQL Server to Oracle replication is performed in a similar manner to that shown earlier for replicating data from Oracle to SQL Server The main difference is that you will create the Extract process groups on the source SQL Server system and the Replicat process groups on the Oracle target system MySQL to Oracle Replication MySQL is a popular RDBMS platform widely implemented by Internet startup companies media companies and Internet Service Providers ISPs as part of the open standard Linux Apache MySQL PHP LAMP technology stack Based on our experiences with Oracle GoldenGate customers they have a need to replicate and migrate their MySQL data as they sca
283. for Windows or UNIX Linux as a direct link or alias to simplify use of the GGSCI command Example C ggs mssqlserver gt ggsci Oracle GoldenGate Command Interpreter for ODBC Version 11 1 1 0 0 Build 078 Windows optimized Microsoft SQL Server on Jul 28 2010 18 55 52 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved HELP The HELP command provides syntax assistance to look up the parameters and details for a specific Oracle GoldenGate command In its most basic form after logging into the GGSCI interface type the HELP command from a GGSCI command shell prompt as shown in the following example GGSCI oracledba 1 gt help GGSCI Command Summary Object Command SUBDIRS CREATE ER INFO KILL LAG SEND STATUS START STATS STOP EXTRACT ADD ALTER CLEANUP DELETE INFO KILL LAG SEND START STATS STATUS STOP EXTTRAIL ADD ALTER DELETE INFO GGSEVT VIEW MANAGER INFO SEND START STOP STATUS MARKER INFO PARAMS EDIT VIEW REPLICAT ADD ALTER CLEANUP DELETE INFO KILL LAG SEND START STATS STATUS STOP REPORT VIEW RMTTRAIL ADD ALTER DELETE INFO TRACETABLE ADD DELETE INFO 293 www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR TRANDATA ADD DELETE INFO CHECKPOINTTABLE ADD DELETE CLEANUP INFO Commands without an object Database DBLOGIN LIST TABLES ENCRYPT PASSWORD DDL DUMPDDL Miscellaneous
284. for the initial load Tip You can set up multiple Extracts and Replicats to handle different sets of tables and run them in parallel to speed up the GoldenGate initial load This is covered in more detail in Chapter 7 First let s look at some of the prerequisites for using the GoldenGate initial load Prerequisites for the GoldenGate Initial Load Before starting the GoldenGate initial load make sure the following tasks have been completed e You must disable any foreign key constraints on the target tables to prevent errors You may also choose to disable check constraints to speed up performance e Disable any triggers on the target tables Triggers firing while the data is being loaded can cause errors www it ebooks info CHAPTER 4 M BASIC REPLICATION e You may want to drop indexes on your target tables to speed up performance This is optional When you ve made sure the prerequisites have been met you can begin setting up the initial load Extract Configuring the Initial Load Extract Let s review the Oracle direct load Extract parameter file used for the example Remember you need to load all the records in the HR schema from the source database into the same schema on the target database You name the new initial load Extract starting with the letter Ito indicate that it s being used for the initial load GGSCI sourceserver 1 edit params IHREMD1 Extract IHREMD1 SETENV 5 LANG AMERICAN AMERICA AL32
285. formed Furthermore you need to use a target definitions file for any column mapping or conversion operations Replicat Errors on the Oracle Database Target System The Oracle Replicat performs the critical task of reading data from the remote trail files on the target database system It uses SQL statements to apply the transactions to the target database system The following issues can occur with Replicat processing and cause failures in Oracle GoldenGate e The Replicat hangs or fails to process data against the target Oracle database e Replicat abends and fails on the target database system e Large transactions cause the Replicat to fail Let s look into each of these issues with Replicats and how to resolve them Replicat Hangs on the Target System Sometimes the Replicat process stalls and doesn t continue to process data from the remote trail file on the target system If this issue occurs check the status of the Replicat process group by issuing the GGSCI command INFO REPLICAT and noting the trail file name as well as the checkpoint information Make sure you verify that the Replicat is reading data from the same trail file the Extract process group is writing to If you discover that this isn t the case then you need to rectify the situation by executing the GGSCI command ALTER REPLICAT group EXTTRAIL trail file name Then to verify that everything is correct issue the GGSCI command INFO RMTTRAIL on the target sys
286. from the current data pump Extract trail and has zero lag showing stop it GGSCI targetserver 1 gt info rep RHREMD1 REPLICAT RHREMD1 Last Started 2011 03 20 13 18 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoint File dirdat 12000013 2011 03 20 13 19 23 000107 RBA 2351 GGSCI targetserver 2 gt stop rep RHREMD1 Sending STOP request to REPLICAT RHREMD1 Request processed Add the new data pump Extract PHREMD2 and tell GoldenGate to start processing at the EXTRBA and EXTSEQ in the Local Extract trail 11 that you recorded in step 5 Be aware that if you don t tell GoldenGate to start at a specific location you can end up re extracting changes from the first trail and cause collisions downstream in the Replicat Run the info command to verify that the Extracts were added properly GGSCI sourceserver gt ADD EXTRACT PHREMD2 EXTTRAILSOURCE dirdat 11 EXTSEQNO 18 EXTRBA 8144 GGSCI sourceserver gt ADD RMTTRAIL dirdat 13 EXTRACT PHREMD2 MEGABYTES 100 GGSCI sourceserver gt info ext PHREMD EXTRACT PHREMD1 Last Started 2011 03 20 13 09 Status STOPPED Checkpoint Lag 00 00 00 updated 00 55 22 ago Log Read Checkpoint File dirdat 11000018 2011 03 20 13 19 23 000000 RBA 8144 EXTRACT PHREMD2 Initialized 2011 03 20 22 12 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 24 ago Log Read Checkpoint File dirdat 11000018 First Record RBA 8144 139 www it ebooks info CHAPTER 7
287. ft SQL Server on the source Windows host For our case study we will be using Microsoft SQL Server 2008 Express Edition with Windows XP 32 bit OS C ggs mssqlserver gt unzip V22241 01 zip Archive V22241 01 zip inflating mgr exe inflating ggsci exe inflating install exe inflating ggMessage dat inflating help txt inflating tcperrs inflating bcrypt txt inflating libxml2 txt Add the manager as a Windows service to simplify things and allow the manager process to run as a background Windows service We recommend this for production environments so that this critical process always runs without stopping C ggs mssqlserver gt install addevents addservice Check the Windows Event viewer to ensure that the Oracle GoldenGate Manager process has been added to Windows as a service as shown in Figure 6 2 File Action View Help es gt aut Services Local Services Local gt uA Ps GGsMGR Description Status Startup Type Log On As ll GameConsoleService GameConsole management services Manual Local Syste Start the service Automatic Local Syste Group Policy Client The service is responsible for applyi Started Automatic Local Syste Figure 6 2 Adding the manager as a Windows Service If the manager process has not been started click the option to start it as shown in Figure 6 3 115 www it ebooks info CHAPTER 6 M HETERO
288. g 277 starting and stopping 63 verifying 63 65 Extract layers 174 309 www it ebooks info INDEX Extract object altering RUN options for 206 Extract parameter configuring for data pump 66 configuring for Local Extract 60 file 71 85 225 Extract processes 54 65 adding and monitoring detail for 156 157 configuring Local Extract 59 62 Extract parameter 60 EXTTRAIL parameter 62 SETENV parameter 61 TABLE parameter 62 USERID parameter 61 62 disabling triggers and cascade delete constraints 57 58 editing Run options 206 error when unable to access database archive and redo logs 235 Extract group adding 63 starting and stopping 63 verifying 63 65 failures 219 221 first time tasks 200 GoldenGate 191 importance of monitoring 155 Local configuring 59 62 310 parallel implementing with table filtering 135 141 overview 134 135 using key ranges 141 146 reporting by 86 89 REPORT parameter 88 89 REPORTCOUNT parameter 88 REPORTROLLOVER parameter 89 source database issues 235 supplemental logging database level 54 55 table level 55 57 verifying Manager status 58 59 Extract report file 231 Extract TABLE parameter 230 extracts limiting number of 288 Extracts and Replicats 287 EXTTRAIL parameter 62 81 EXTTRAILSOURCE parameter 67 F failures fetch 239 240 Replicat process Abend failures 236 237 on large transactions 238 www it ebooks info
289. g parallel Extracts and Replicats using key ranges for your HR schema Implementing Parallel Extracts and Replicats Using Key Ranges Another strategy for creating parallel Extracts and Replicats is to use the GRANGE function to split the incoming table rows into equal buckets using a GoldenGate hash algorithm on the key values Using this technique you can create even more parallel processes than you can just by splitting using table filtering You do this to divide the replication workload into parallel processes by key ranges and make the replication process more data and reduce the Replicat lag Figure 7 4 shows more details about how you split the Replicat using key ranges for the HR schema 141 www it ebooks info CHAPTER 7 88 TUNING 142 ec Range 1 3 Ce RHREMD1 Local Source Data Remote W Replicat 1 Source IN Extract D Trail ii Pumpt Trail 1 D RHREMD1 Target Database LHREMD1 qq Sudan 12 Range 2 3 Database Replicat 1 RHREMD1 Range 3 3 Figure 7 4 Parallel Replicat groups split by key range Local Extract LHREMDI remains unchanged and still extracts all the tables for the HR schema Using the Local Extract to extract all the data is a common approach In this example you also leave the data pump unchanged to pass through the complete trail from the Local Extract You could split the trails using the data pump but this section shows another technique and splits processing using the Repl
290. g the Local Extract to extract all the data is a common approach Typically it s better to use the Local Extract to extract all o the data and then use the data pump Extracts to do any filtering Data Pump PHREMDI is changed from extracting all the tables in the HR schema to filtering only for HR tables EMPLOYEES JOB HISTORY JOBS and LOCATIONS The other tables in the schema COUNTRIES and REGIONS are extracted a new data pump Extract PHREMD2 Next you use two Replicat groups to process the two remote trail files from the data pumps Tip To avoid contention Oracle recommends that when reading trail files you pair each Replicat with its own trail file No more than three Replicats should be reading the same trail file Replicat RHREMDI processes the data pumped from data pump PHREMDI in the 12 trail file The new Replicat RHREMDI processes the data pumped from data pump 2 in the new 13 trail file Next let s go over the steps to implement your tuning changes Because you already have replication running for the HR schema you have to make the tuning changes to the existing running configuration as shown in Figure 7 1 Following are the steps to add the parallel data pump Extract and Replicat to the existing replication configuration As a reminder although it isn t shown here so you can see the command details you should put the GoldenGate commands in these steps in an obey file as covered in Chapter 4 Tip Al
291. ggestions and you should collect any metrics that can have an influence on the performance of your particular replication environment Collect the metrics for at least for one hour You can collect them for a week or even a month if that makes sense for your particular environment You can also capture metrics for special situations For example you may have some month end processing that causes your replication volume to increase You can capture a baseline for this activity when you consider it to be running normally and then use the baseline to compare to subsequent months Let s assume that you ve done a good job of collecting your baseline statistics and a few weeks later you get a report that the replication is running slowly That triggers you to move to the next step in your process evaluating the current performance Evaluating the Current Performance When a potential GoldenGate replication performance problem has been reported the next step in the process is to evaluate the current performance against the baseline that you recorded earlier You should follow the exact same steps to collect the current statistics during the performance problem conditions as when you collected the baseline performance statistics under normal conditions Let s look at a sample of some baseline and current processing statistics collected for HR Extracts and Replicats as shown in Table 7 1 Table 7 1 Extract and Replicat Performance Statistics Stat
292. gging on the source database Remember that if you re using bidirectional replication you need to enable supplemental logging on both the source and the target databases The supplemental logging is required to ensure that Oracle adds additional information to the redo logs that is required by GoldenGate First you can verify whether Oracle database supplemental logging is already enabled by running the following SQLPLUS command SOL SELECT SUPPLEMENTAL LOG DATA MIN FROM V DATABASE SUPPLEMENTAL LOG DATA MIN For Oracle 9i SUPPLEMENTAL LOG DATA MIN must be YES For Oracle 10g and later SUPPLEMENTAL _ LOG DATA must be YES or IMPLICIT Now that you ve verified that database level supplemental logging isn t currently enabled you can enter the commands to make it enabled Enabling Database Level Supplemental Logging You can enable Oracle database level supplemental logging by running the following SQLPLUS command as a user with the ALTER SYSTEM privilege 54 www it ebooks info CHAPTER 4 M BASIC REPLICATION SQL gt ALTER DATABASE ADD SUPPLEMENTAL LOG DATA Verify that it is enabled SOL SELECT SUPPLEMENTAL LOG DATA MIN FROM V DATABASE SUPPLEMENTAL LOG DATA MIN Remember for Oracle 9i SUPPLEMENTAL LOG DATA must be YES For Oracle 10g and later SUPPLEMENTAL LOG DATA must be YES or IMPLICIT Enabling Table Level Supplemental Logging Normally databases such as Oracle DB2 and SQL Server don t log enough data in the tran
293. ggs ggserr log Finally you can run the stats command on your Extract This shows whether the Extract has actually processed any DML changes In the example the Extract has processed four inserts and two deletes since it was started as shown here GGSCI sourceserver 2 stats ext LHREMD1 Sending STATS request to EXTRACT LHREMD1 Start of Statistics at 2011 01 18 18 50 38 Output to gger ggs dirdat 11 Extracting from HR EMPLOYEES to HR EMPLOYEES Total statistics since 2011 01 15 13 54 52 Total inserts 4 00 Total updates 0 00 Total deletes 2 00 Total discards 0 00 Total operations 6 00 You re now finished starting the Local Extract so let s move on to the next step starting the data pump Extract Starting the Data Pump Now that the Local Extract is started you can proceed with configuring adding and starting the data pump First let s refresh your memory a bit on the purpose of the data pump from Chapter 3 The data pump is another type of GoldenGate Extract process It reads the records in the source trail written by the Local Extract pumps or passes them over the network to the target and creates a target or remote trail In the example you configure the data pump to read the 11 trail file written out by the Local Extract named LHREMDI and pump it over the TCP IP network to the target server to be processed by the Replicat Configuring the Data Pump From GGSCI let s edit the parameters for the data pu
294. gh This reduces the network traffic from Veridata Agent to Veridata Server Comparison methods can be changed in the Column Mapping screen 173 www it ebooks info CHAPTER 9 88 ORACLE GOLDENGATE VERIDATA How Veridata Can Help You You can use Veridata in so many different ways it is impossible for me to list all of them here The following are some of the key features and benefits 174 It works seamlessly with data replication tools such as GoldenGate one way replication and two way synchronization GoldenGate just captures and delivers the data but it won t check the data for you unless you add some custom codes to do the validations Veridata can do two phase comparison that will report the data discrepancies on the bad data and any data that GoldenGate skips Enhances enterprise data quality among all the systems where bad data can lead to poor decision making failed service level agreements and ultimately operational financial and legal risk With Veridata validating all your business critical data constantly you can identify the bad data before business users report it Veridata supports Oracle materialized views views and Veridata specific partition that make it feasible to compare the data in different granularities and cardinalities Reduces time in data comparison because of the high performance Veridata optimizes the comparison in Extract layers in the Agent by using hashed values then it compares the data in a
295. ginal replication configuration from the primary to the secondary database Let s review those steps in part 5 as shown in Figure 12 13 Start the Extract Start the Pump Start the Replicat 1 4 TAA I Standb Primary gt Local vt D Database Data pump E Replicat Database kal VO VO cn Database Application ag 8 Start the Application 2 Figure 12 13 Disaster recovery unplanned failover part 5 Following is a description of the steps in part 5 as shown in Figure 12 13 1 To begin capturing changes in preparation for the application switchback to the primary database alter the Local Extract for the primary database to begin capturing changes beginning now and then start the Extract as follows GGSCI primaryserver gt ALTER EXTRACT LHREMD2 BEGIN NOW GGSCI primaryserver gt START EXTRACT LHREMD2 2 Prepare the primary database for the application and start the application connected to the primary database In preparation you may need to grant some SQL permissions to your application users Also remember to enable any triggers or cascade delete constraints you may have stopped while the primary database was the replication target 3 Start the data pump Extract on the primary database as shown in the following example GGSCI primaryserver gt START EXTRACT PHREMD2 261 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION 262 4 Start the the Re
296. group You can use the SPECIALRUN parameter to let GoldenGate know that this is a special Replicat to be used only for the initial load Later this chapter covers setting up another Replicat for ongoing synchronization Now that you have the initial load Extract and Replicat configured and added to GoldenGate you can start the load process Starting the GoldenGate Initial Load You start the GoldenGate initial load process by starting the Extract Remember it isn t necessary to start the Replicat because GoldenGate does this automatically for the initial load when the initial load Extract is started GoldenGate only automatically starts the Replicat for an initial load Replicat not for the ongoing change Replicat When the load is complete GoldenGate automatically stops the initial load Extract and Replicat Let s start the IHREMDI Extract as shown in the following example GGSCI sourceserver gt START EXTRACT IHREMD1 If needed for some reason you can stop the initial load Extract using a similar method For example you may need to make a change to the GoldenGate parameters on the initial load Extract You stop and then start the initial load Extract for the new changes to take effect You must remember to check whether any rows have already been loaded to the target table If so you need to delete those rows on the target before restarting the extract Following is an example of stopping the IHREMDI Extract GGSCI sourceserver gt
297. guration If you allow writes to the standby then you need to handle any data conflicts as discussed in Chapter 5 To allow for the fastest recovery time typically you will want the database structures and data in the standby database to be identical to the primary database However by using GoldenGate you have the flexibility to allow differences in your primary and standby databases For example you may decide you only want to keep a critical subset of your data for disaster recovery purposes rather than the entire database You can do this with GoldenGate filtering as covered in Chapter 5 For Oracle databases you can also use the GoldenGate DDL replication feature to keep the database tables synchronized as well as the data Usually you will want the data in the primary database to be kept current with the disaster recovery site to within at most a few seconds of difference in case of failover For example if you were processing financial transactions any lag differences between the primary and the standby data could be costly This will vary depending on your particular application For some applications you may be able to tolerate the standby lagging behind the primary database for a period of time Also there may be some databases which do not need to be kept current and only need to be refreshed periodically In that case you can still set up the same GoldenGate configuration we will cover in this chapter but only turn it on as needed
298. gured The most important performance requirement for replication is the amount oflag the application can tolerate As you gather your replication requirements you should think of lag in terms of the overall delay in replicating from the source to the target Later you ll break the lag down into specific Extract and Replicat lag for the GoldenGate environment In Chapter 4 you looked at some of the requirements that drive your replication technical design The following two areas relate specifically to performance and tuning of replication Data currency How quickly and how often does the data need to be updated Can any lag in the data replication be tolerated If no lag can be tolerated and there is a high volume of changes you may need to devote more time in the project to designing and tuning the replication to avoid any lag Keep in mind that often reporting systems can tolerate lag and the target data doesn t need to be completely up to date with the source Data volume How much data needs to be replicated How often is it updated You may be able to examine the existing database transaction log files to determine the amount of data change that is occurring in the source database The data change volume impacts the network bandwidth requirements between the source and target and the amount of disk space required for trail files After you ve worked with your business users and stakeholders to answer these questions you can begin to f
299. guring the Fallback Local Extract for Zero Downtime Migration 272 Configuring the Fallback Data Pump for Zero Downtime Migration e 272 Configuring the Fallback Replicat for Zero Downtime Migration eene 273 Performing the Migration Cutover eeeeeeeeeeneenn eren nnn nnn 274 Performing the Migration Fallback 1 eeereere eren enne nnn nnn 276 ii M 278 Chapter 14 Tips and 279 Requirements PLANNING 279 Knowing the Business 279 Understanding the 280 Determining the Topology erre ere enses ene 281 installation SOW EDO DK Or rx 281 Creating Dedicated 5 5
300. guring the Local Extract Now that you ve made sure the Manager is running let s configure the Local Extract In order to do this you first need to create a parameter file for the Extract Remember for now as the example you re configuring the local Extract to capture all the SQL DML changes from the sample HR schema Tip You can edit the GoldenGate parameters directly from the GGSCI or using the native OS text editor outside of GGSCI whichever is most convenient To edit in GGSCI you enter EDIT PARAMS paramterfilename GGSCI brings up the parameter file in the default text editor which is Notepad for Windows and vi for UNIX and Linux You can change the default editor using the SET EDITOR command in GGSCI If you re editing directly from the OS save the parameter file in the dirprm subdirectory under the GoldenGate software installation directory In the example the parameter file directory is gger ggs dirprm Let s start by looking at the parameters for the Local Extract as shown in the following example GGSCI sourceserver 1 edit params LHREMD1 Extract LHREMD1 59 www it ebooks info CHAPTER 4 M BASIC REPLICATION SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER SourceDB PASSWORD userpw ExtTrail dirdat l1 Table HR The parameters and terminology may seem a little strange to you at first but over time you get used to the format and usage The next sections look at each of the parameters in more
301. h certain limitations Be sure to check the GoldenGate product documentation for a list of data types supported Security requirements How much security is required for the replication For example is it necessary to encrypt the data that GoldenGate is replicating GoldenGate can encrypt both the data stored in trail files as well as the data being sent from the source to the target Also is the data in the database encrypted For example Oracle can encrypt database data using a feature called Transparent Data Encryption TDE If the database is using TDE GoldenGate needs special parameters defined to unencrypt the data for replication 50 www it ebooks info CHAPTER 4 M BASIC REPLICATION Network requirements Describe the network between the source and the target Are there any firewalls that need to be opened for the replication What is the distance between the source and target How much bandwidth is required to handle the data volume and how much is available Table 4 1 describes the environment used for discussion purposes in this chapter Noted in parentheses are some settings for Windows with a SQL Server database environment Remember though the sources and targets can be any platform and database supported by GoldenGate as reviewed in Chapter 3 You may want to set up a table like this for your replication projects Table 4 1 Sample GoldenGate Replication Environment Configuration Item Value GoldenGate soft
302. has caught up and applied all the changes the databases are synchronized 53 www it ebooks info CHAPTER 4 M BASIC REPLICATION Note These steps assume a zero downtime replication configuration Online DML changes will be occurring in the source database while the initial load process is running This is the most likely scenario in today s environments where downtime usually needs to be avoided If you can quiesce or stop DML activity against the source database while the initial load is occurring it is not necessary to start the Extract before the initial load Let s look at each of the basic replication steps in detail beginning with setting up and starting the Extract Starting the Extract When you ve verified that the prerequisites are complete the first step you need to complete in setting up replication is to start the Extract Remember that you should start the Extract first to begin capturing changes made while the initial data load is running If you can afford to take an outage and stop all change activity on the source database while the initial load is running then you can start the Extract after the initial load This section covers the various Extract configuration settings and options After that you see how to start the Extract and verify that it s properly capturing changes Verifying Database Level Supplemental Logging When replicating from Oracle databases you need to enable database level supplemental lo
303. he Local Extract Because you re replicating the PAYROLL database you can use PR as the abbreviation for PAYROLL in the Local Extract name You can use EM to indicate that the PAYROLL database is part of the employee application You exclude the two tables that don t need to be replicated EXCLUDETABLE1 and EXCLUDETABLE2 with the TableExclude parameter Notice that you re replicating all of the tables in the PAYROLL database using the asterisk as a wildcard and then excluding onlythe two tables you don t want to replicate Now that you ve set up the Extract configuration parameters on the old server the next step is to add the Extract group You can do that using the following commands from the GoldenGate Software Command Interface GGSCD GGSCI oldserver gt ADD EXTRACT LPREMP1 TRANLOG BEGIN NOW GGSCI oldserver gt ADD EXTTRAIL dirdat p1 EXTRACT LPREMP1 MEGABYTES 500 You can use p1 as the Local Extract trail file name Because the payroll application has a high volume of data changes you make the trail file size 500MB After adding the Extract you need to start it to actually begin capturing changes as shown in the following example GGSCI oldserver START EXTRACT LPREMP1 Next let s check to make sure the Extract is running properly using the INFO command GGSCI oldserver 2 INFO EXTRACT LPREMP1 If the Extract isn t running you can review the Oracle GoldenGate error log and report file and try to resolve the problem w
304. he SQL to fetch and return blocks of data to the Server to compare It also extracts the out of sync rows detail to the Server for reporting purposes Java agent doesn t require GoldenGate Manager but C code based agent requires GoldenGate Manager Process GoldenGate Veridata CLI Vericom Vericom is a command line interface to schedule the compare jobs It can override some profiles settings partition sql predicates shut down Veridata Server and generate review the out of sync reports However it cannot configure the comparison settings and rules that only can be done on the Veridata Web client How Veridata Comparison Works By default Veridata comparison activities are divided into two steps initial compares and confirmation compares Confirmation compare is optional if the data is not expected to change during the compare or the data is filtered by the partition sql predicates Initial compare generates an Out Of Sync report OOS The Confirmed Out Of Sync COOS reports are created during the confirmation compare process By default Veridata compares rows literally value for value for all primary key columns It uses a hash value for all non key columns to compare The row hash value is a compressed unique digital signature that is used to compare two source values For example a row length of 2000 bytes can be transposed to 12 bytes hashed value It is not absolute but the likelihood that two original values are identical is very hi
305. he target Microsoft SQL Server system as shown in Figure 6 8 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION Create a New Data Source to SQL Server CX This wizard will help you create an ODBC data source that you can use to connect to SQL Server What name do you want to use to referto the data source Name ggstosqlsrv How do you want to describe the data source Description GGS SQL server Which SQL Server do you want to connect to Server ORACLEDBA b Figure 6 8 Add New Data Source for the Microsoft SQL Server database Now that we have finished the initial setups on both source and target environments we are ready to set up the parameter file configurations for Oracle and Microsoft SQL Server to use as part of data replication with Oracle GoldenGate Configure Change Data Capture on Source Next you will need to configure the Change Capture Process on the Oracle Source database system GGSCI ADD EXTRACT EXTORA TRANLOG BEGIN NOW EXTRACT added GGSCI INFO EXTRACT EXTORA GGSCI oracledba 4 info extract extora EXTRACT EXTORA Initialized 2011 04 03 23 15 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 31 ago VAM Read Checkpoint 2011 04 03 23 15 12 005000 The next step is to create and associate the trail files with our extract from the previous step Using GGSCI and the ADD RMTTRAIL commands add the trail file as shown below 121 www it ebooks in
306. icat processes require access to in order to send and receive data This range of ports must be allocated for the Goldengate environments Also allocate ports for Goldengate manager Extract and Replicat processes By default manager uses port 7840 We recommend that you keep this port available In addition keep a record of ports allocated to Goldengate processes to avoid port conflicts 17 www it ebooks info CHAPTER 2 INSTALLATION Operating System Requirements Following are some requirements when running under Windows You must use the Administrator account to install the Oracle Goldengate software You must Install the Microsoft Visual C 2005 SP1 Redistributable Package You must use SP1 for these libraries You can obtain the correct version for your specific Windows platform at www microsoft com Under Linux or UNIX you should do the following Grant read and write privileges for the operating system OS account used to install the Oracle Goldengate software Place the Oracle Goldengate software on a shared disk in a clustered environment or on a shared clustered filesystem that all cluster nodes have access to Install from an operating and database system account that has read write access to Oracle database software as well as to the Oracle online redo log files Note For Itanium you must install the vcredist IA64 exe runtime library package which provides the mandatory Visual Studio DLL libraries require
307. ication Clusters RAC Later in the chapter you expand the basic one way replication into a bidirectional replication topology Enhancing the Replication Configuration This section covers some recommended enhancements to your replication configuration It begins with enhancements to the basic reporting configuration Next it covers the purging of old trail files and adding restartability to your Extracts and Replicats Finally you see how to add a checkpoint table in your database to make the replication more robust Before you get started let s review the basic replication configuration parameters from Chapter 4 First here s the basic Extract parameter file GGSCI sourceserver 1 edit params LHREMD1 Extract LHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER SourceDB PASSWORD userpw ExtTrail gger ggs dirdat 11 Table HR Next is the basic Replicat parameter file GGSCI targetserver 1 gt edit params RHREMD1 Replicat RHREMD1 www it ebooks info 85 CHAPTER 5 ADVANCED FEATURES SETENV NLS_LANG AMERICAN AMERICA AL32UTF8 USERID GGER TargetDB PASSWORD userpw HandleCollisions AssumeTargetDefs Map HR Target HR Finally the Manager parameter file is as follows GGSCI sourceserver 1 edit params MGR port 7840 You expand on these basic parameters throughout this chapter and see where to make changes and additions as you go along Now let s move on to enhancing your reporting
308. icats Replicat RHREMDI is split into three Replicats The first Replicat RHREMDI processes the first range of keys The second Replicat RHREMD2 processes the second range of keys and so on You use the GoldenGate GRANGE function to split the keys into three buckets using a built in hashing algorithm Now let s go over the steps to implement your tuning changes Again this example assumes you re implementing the tuning changes to the basic one way replication topology shown in Figure 7 1 Following are the steps to add the parallel data pump Extract and Replicat to your replication configuration As a reminder this list doesn t show the commands in an obey file so you can see the command details but you should put the GoldenGate commands in these steps in an obey file as covered in Chapter 4 1 Edit the existing Replicat parameter file RHREMDI to add the RANGE filtering GGSCI targetserver gt edit params RHREMD1 Replicat RHREMD1 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY Map HR EMPLOYEES Target HR EMPLOYEES COLMAP USEDEFAULTS FILTER RANGE 1 3 Map HR JOBS Target HR JOBS COLMAP USEDEFAULTS FILTER RANGE 1 3 HR JOB HISTORY Target HR JOB HISTORY COLMAP USEDEFAULTS FILTER QRANGE 1 3 Map HR LOCA
309. ile e DOWNREPORT Reports on down processes every user specified number of hours or minutes e DOWNCRITICAL Writes a critical message if the Extract or Replicat fails e LAGCRITICAL Writes a warning message when the user specified time period is exceeded e LAGINFO Tells Manager to report lag info every user specified time period www it ebooks info CHAPTER 14 8 TIPS AND TRICKS Purging Old Trail Files Regularly Old trail files can accumulate and cause disk space issues Trail files can usually be purged after a couple days but they can be kept for more or less time depending on your requirements and the amount of available disk space You can specify automatic purge criteria in the Manager parameter file as shown in the next example Keep in mind that specifying the purge criteria using Manager applies to any Extracts and Replicats running under the Manager This allows you to manage the purging criteria centrally The following example tells Manager to purge any trail files older than two days The UseCheckPoints option tells Manager not to purge the files until after they have been processed as shown in the following example PurgeOldExtracts dirdat UseCheckPoints MinKeepDays 2 Next let s look at some tips for automatically starting your GoldenGate processes Automatically Starting Processes Use the AUTOSTART Manager parameter to have the GoldenGate Manager automatically start your Extract and Replicat processes
310. in Sybase source database e Quiesce the Sybase RepServer before starting GoldenGate Extract process You need to do so because Oracle GoldenGate Extract process uses Sybase LTM to read from the Sybase transaction log e Grant permission to Oracle GoldenGate Extract process to allow the ability to manage secondary log truncation point e The Sybase source database must be an active server It cannot be in warm standby mode Further details with respect to the Sybase and GoldenGate installation procedures are available in the Oracle GoldenGate Sybase Installation and Setup Guide 115 Release 1 11 1 1 available online at http download oracle com docs cd E18101_01 index htm Installing GoldenGate for IBM DB2 UDB on Windows and UNIX We will focus on the database requirements for the Oracle 11g GoldenGate installation with IBM DB2 UDB database with Windows and UNIX platforms Disk space storage requirements are the same as for general Oracle GoldenGate installation as already discussed In addition you will need to grant read and write permissions at both the database and operating system level for the Oracle GoldenGate user accounts on source and target DB2 systems with GoldenGate The following items must be configured as part of the DB2 and GoldenGate installation e Read and write permissions at database and operating system level to access the DB2READLOG functions so that the Oracle GoldenGate Extract process can read the DB2 UDB tran
311. in the following example for Oracle on Solaris To find out the GoldenGate version change to the GoldenGate home directory and issue the GGSCI v command from the operating system terminal shell window C Mggs src ggsci v Oracle GoldenGate Command Interpreter for Oracle Version 11 1 1 0 0 Build 078 Windows optimized Oracle 11 on Jul 28 2010 17 20 29 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved Database Availability Issues Ifthe source and target database environments aren t online and running correctly Oracle GoldenGate processing fails because the Extract and Replicat processes frequently log in to the database environments to access redo log files for processing data replication activities You can check whether the database and listener are online by using the LSNRCTL and TNSPING commands for Oracle from an operating system shell window C Nggs src lsnrctl status LSNRCTL for 32 bit Windows Version 11 2 0 1 0 Production on 14 MAR 2011 01 53 Connecting to DESCRIPTION ADDRESS PROTOCOL IPC KEY EXTPROC1521 STATUS of the LISTENER Listening Endpoints Summary DESCRIPTION ADDRESS PROTOCOL ipc PIPENAME W 1521 DESCRIPTION ADDRESS PROTOCOL HOST oracledba PORT 1521 Services Summary Service CLRExtProc has 1 instance s Instance CLRExtProc status UNKNOWN has 1 handler s for this service Service winiisrc has 1 instance s I
312. inanan ananena naanin naaa aana 287 XV www it ebooks info CONTENTS POMTORMANCE e 287 Running Performance Tests 287 Limiting the Number of 288 Using Passthru Mode for Data PUMPS sssssssssesssssesssssssessessessessssssssseeseesssnssessesseeseeneenssssesseens 288 Using Parallel RepliCats eene tnnt tnnt nenne nnns 288 Using the Fastest Available Storage essssssseeseeeen nennen tnn nnns 288 Tuning the Database nennen tnnt 289 289 Appendix Additional Technical Resources for the Oracle GoldenGate Administrator ener 291 References for Further Reading neon canas 291 Quick Guide to Oracle GoldenGate Commands ees 292 rape M 292 293 293 INFO M 294 295 STATUS 295 Logdump Commands and Syntax for Troubleshooting
313. ind the next record with timestamp lt date time string gt lt filename template gt lt date time string gt format is yy yy mm dd 551 SCANFORENDTRANS SFET Find the end of the current transaction SCANFORNEXTTRANS SFNT Find start of the next transaction SHOW lt option gt Display internal information OPEN list open files TIME print current time in various formats ENV show current environment RECTYPE show list of record types FILTER show active filter items BIO option Set LargeBlock I O info ON Enable LargeBlock 1 0 default OFF Disable LargeBlock 1 0 BLOCK lt nnnn gt Set LargeBlock 1 0 size TIMEOFFSET lt option gt Set the time offset from GMT LOCAL Use local time GMT Use GMT time GMT hh mm Offset from GMT FILTER SHOW FILTER ENABLE ON Enable filtering FILTER DISABLE OFF Disable filtering FILTER CLEAR filterid ALL FILTER MATCH ANY ALL FILTER INClude EXCLude filter options filter options are RECTYPE type number type name STRING BOTH text column range HEX hex string lt column range TRANSID TMF transaction identifier FILENAME filename template PROCESS lt processname template 297 www it ebooks info APPENDIX ADDITIONAL TECHNICAL RESOURCES FOR THE ORACLE GOLDENGATE ADMINISTRATOR INT16 lt 16 bit integer gt INT32 lt 32 bit integer gt INT64 64 bit int
314. iness can be performed by using scripts to generate initial loads perform DML and DDL transactions such as updates deletes and inserts as well as using Oracle GoldenGate Veridata to check for data synchronization status between source and target heterogeneous environments with Oracle GoldenGate Another way to measure operational readiness for your Oracle GoldenGate environments is to use the Oracle GoldenGate Director product to check on status for replication between source and target environments 124 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION Summary We provided a case study walkthrough on heterogeneous real time data replication to Oracle from third party database platforms with Oracle GoldenGate We began with some basic installation guidelines and provided sample configuration steps along with the required commands to get up and running with minimal time and effort In a large production environment with complex transformations you would most likely need to build out these simple examples with more advanced features such as tokens and filters along with macro libraries to enhance the business requirements of your replication environment 125 www it ebooks info 7 Tuning GoldenGate is a proven high performance product that replicates data for some of the busiest high volume applications in the world To make GoldenGate perform efficiently in these types of environments you need to have a
315. ing 68 verifying 68 70 www it ebooks info DBMS configuration options 99 106 for ASM 102 103 DDL replication 103 106 for RAC 100 102 enhancing configuration 85 92 automatic process startup and restart 90 checkpoint table 90 92 purging old trail files 89 90 reporting 86 89 Extract process 54 65 configuring Local Extract 59 62 disabling triggers and cascade delete constraints 57 58 Extract group 63 65 supplemental logging 54 57 verifying Manager status 58 59 and GoldenGate 4 heterogeneous See heterogeneous replication integration 44 45 loading 70 75 with DBMS utilities 76 initial 70 75 one way 40 42 200 203 overview 49 54 basic steps 52 54 one way replication topology 52 INDEX prerequisites for setting up 49 50 requirements 50 52 Replicat process 76 83 adding 81 configuring 77 81 starting and stopping 81 verifying 82 83 security 92 94 servers 22 streams replication 3 4 topology one way 52 for zero downtime migration 263 278 cutover 274 276 fallback 276 278 prerequisites 263 264 requirements 264 265 setup 266 274 topology 265 266 report diagnostics process failures without 221 Report link 184 REPORT parameter 88 89 REPORTCOUNT parameter 88 reporting discarded records 89 by Extract and Replicat processes 86 89 REPORT parameter 88 89 323 www it ebooks info INDEX 324 REPORTCOUNT parameter 88 REPORTROLLO
316. ing is a sample Extract parameter file EXTRACT exti SETENV ORACLE HOME oracle ora11g product SETENV ORACLE SID oraiisrc www it ebooks info CHAPTER 2 88 INSTALLATION USERID ggs PASSWORD ggs RMTHOST sdtarget RMTTRAIL d1 oracle ggs dirdat rt TABLE scott emp For Windows servers that have multiple Oracle database instances you can set the environment variable for ORACLE HOME and ORACLE SID by adding these settings to the registry for Windows This can be performed by opening My Computer Settings Properties Advanced tab then Environment Variables and System Variables GoldenGate and Oracle RAC Considerations Oracle RAC is certified by Oracle to support Goldengate software However you will need to keep the following things in mind when performing installation for Oracle Goldengate with RAC environments e All GoldenGate software binary and executable files trail files and log files should be on shared storage accessible by all cluster nodes e System clocks must be synchronized for all RAC nodes using the Goldengate software You can set up the network time protocol NTP to ensure that cluster nodes are kept in sync In addition you can use the Goldengate IOLATENCY option with the THREADOPTIONS parameter for Extract and Replicat parameter files Installing Goldengate for Microsoft SQL Server on Windows The Oracle GoldenGate installation process for Microsoft SQL Server requires additional configuration set
317. ing of supplemental redo data enabled for table HR EMPLOYEES The DBLOGIN command establishes a database connection for the gger database user The ADD TRANDATA command causes the database to log primary key columns for all update operations on the HR EMPLOYEES table ADD TRANDATA performs different database commands in the background depending on the database Behind the scenes the ADD TRANDATA command is equivalent to the following command in an Oracle database 55 www it ebooks info CHAPTER 4 M BASIC REPLICATION 56 ALTER TABLE HR EMPLOYEES ADD SUPPLEMENTAL LOG GROUP GGS EMPLOYEES 19387 EMPLOYEE ID ALWAYS For SQL Server 2008 you notice that ADD TRANDATA enables Change Data Capture CDC and sets up two new jobs to allow the GoldenGate Extract to use CDC GoldenGate will minimize the amount of CDC data by using primary key or the smallest columns available You should verify the supplemental logging was successfully enabled with an Oracle query such as the following SOL select owner log group name table name from dba log groups where owner HR OWNER LOG GROUP NAME TABLE NAME HR GGS EMPLOYEES 19387 EMPLOYEES If you have a large number of tables it may be time consuming to enter all the ADD TRANDATA commands individually As an alternative you can use a script such as the following Oracle SQLPLUS script to automatically generate the ADD TRANDATA commands for a large number of tables set echo off set verify
318. ing up 244 251 data pump 246 247 Local Extract 245 246 Replicat 247 248 standby data pump 249 250 standby Extract 248 249 standby Replicat 250 251 topology of 243 for unplanned failover 256 www it ebooks info discard files 223 225 286 missing 224 not opening 225 too big 224 DISCARDFILE parameter 89 223 226 DISCARDROLLOVER parameter 89 224 disks requirements installation of Oracle GoldenGate for Teradata on Microsoft Windows and UNIX operating system 22 Oracle GoldenGate Veridata 28 29 space on 17 168 169 storage 149 150 distributed processing 1 2 DNS domain name server 231 DOWNCRITICAL parameter 286 downloading software 5 12 from Oracle e delivery 5 8 from OTN 9 12 DOWNREPORT parameter 286 DYNAMICPORTLIST parameter 232 E Edit Database Connection setting Truncate Trailing Spaces When Comparing Values option 181 Edit Param feature 209 INDEX EDIT PARAMS parameter file name 59 77 editors modifying Manager parameter file using 193 EM abbreviation 268 encrypting passwords 92 282 trail files 93 94 ENCRYPTTRAIL parameter 93 env command 230 envigrep ORA command 231 environment 12 14 error logs 162 223 error message 501 226 errors missing column 239 Replicat process 236 238 event logs 162 event viewer Windows 116 Exit codes 186 Extract and Replicat parameter files 286 Extract checkpoints 232 234 Extract group 268 adding 63 startin
319. istener ora file SID DESC SID NAME ASM1 ORACLE HOME u01 asm oracle product 11 1 0 asm 1 This clause ensures that your listener is listening for connections on the ASM instance Make sure to reload your listener for the changes to take effect Updating the TNSNAMES ORA Finally you need an alias in your tnsnames ora file for the ASM connection ASM DESCRIPTION ADDRESS PROTOCOL TCP HOST sourceserveri vip PORT 1521 CONNECT DATA SERVER DEDICATED 102 www it ebooks info CHAPTER 5 ADVANCED FEATURES SERVICE_NAME ASM1 INSTANCE NAME ASM1 The entry should match the alias you specified earlier in your GoldenGate ASMUser parameter The example uses an ASMUser of sys asm to connect in the GoldenGate parameter file which matches your TNS entry Adding Oracle DDL Replication In addition to replicating DML statements GoldenGate can replicate DDL statements for all Oracle database releases that support DML replication You can configure GoldenGate to replicate only DDL only DML or both DDL and DML statements with the same Extract and Replicat parameter file Before you jump into configuring the parameters for DDL replication you should be aware of some of the major restrictions with GoldenGate DDL replication You must turn off the Oracle recycle bin in Oracle 10g and later prior to Oracle 11281 GoldenGate can only automatically replicate DDL statements that are less
320. istics Baseline Current Source CPU Utilization 12 15 Target CPU Utilization 17 73 Source Disk Reads sec 2 1 2 9 Source Disk Writes sec 10 1 10 6 Source Disk Service Time 3 4 3 2 Target Disk Reads sec 2 1 2 8 Target Disk Writes sec 10 1 10 9 Target Disk Service Time 3 4 3 2 Local Extract LHREMDI Status RUNNING RUNNING Data Pump PHREMDI Status RUNNING RUNNING Replicat RHREMDI Status RUNNING RUNNING 131 www it ebooks info CHAPTER 7 88 TUNING Continued Statistics Baseline Current Local Extract LHREMD1 Rate Delta 20 21 20 19 Data Pump PHREMDI Rate Delta 20 21 20 19 Replicat RHREMDI Rate Delta 15 16 9 6 Replicat RHREMDI Operations Min 1948 1232 Local Extract LHREMDI Secs Lag 12 11 Data Pump PHREMDI Sec Lag 8 9 Replicat RHREMDI Secs Lag 15 127 You probably notice some major differences when comparing the baseline statistics to the current performance statistics They may indicate a performance problem but you should be careful not to jump to conclusions Let s move on to the next step in the tuning process to determine the problem Determining the Problem After you ve collected the current performance statistics you can move on to try to determine the actual performance problem Even though you re now armed with plenty of statistics there are still many unanswered questions Let s look at some of the required tasks when going through your analysis to determine the problem 1 Analyze the baseline statistics
321. ite this book I d also like to thank Jonathan Gennick Anita Castro Douglas Pundick and everyone at Apress for their help and advice in writing the book Also thanks to my colleague Sherri Trojan for her kind support and encouragement Finally thanks to my friend Chris Giametta who got me started on this adventure Steve Phillips My thanks to the technical review team especially Arup Nanda who gave me the opportunity to write this book and provided many good suggestions and comments in his technical review Also thanks to the team at Apress for continuously working with me to improve this book Richard Chung www it ebooks info
322. ks info for Teradata on Windows and UNIX 22 for Oracle GoldenGate Director 11g 25 26 client installation requirements 26 software requirements 25 26 supported platforms 25 web client installation guidelines 26 for Oracle GoldenGate Veridata 29 31 database privileges 30 disk requirements 29 memory requirements 29 software requirements 29 30 web requirements 30 31 for Oracle GoldenGate Veridata agent 28 T Table EMPLOYEES 52 table filtering implementing parallel Extract and Replicat processes with 135 141 TABLE parameter configuring for data pump 67 configuring for Local Extract 62 TABLE statements 61 96 239 TableExclude parameter 268 table key constraints 239 INDEX table level supplemental logging enabling 55 57 tables 176 177 checkpoint 282 comparing incremental data for large 181 182 filtering 94 95 missing key constraints 239 tail command 223 tar command 20 target databases 39 40 target definitions file 236 TargetDB 73 78 TCP IP error 231 TCP IP network connection 38 TCP IP network latency 221 TCP IP socket buffer sizes 38 TDE Transparent Data Encryption 281 Teradata 12 0 123 Teradata 13 0 123 Teradata RDBMS relational database management system installation of Oracle GoldenGate 21 22 to Oracle replication 123 tests performance 287 THREADOPTIONS IOLATENCY 100 327 www it ebooks info INDEX THREADOPTIONS MAXCOMMIT
323. l 54 55 table level 55 57 logs checking archived 157 158 current online redo 157 event and error 162 redo monitoring consumption rate 158 159 Low latency 109 LSNRCTL command 229 management and monitoring 284 287 automatically starting processes 287 discard file 286 generating interim statistics 286 GGSCI command shortcuts 284 285 OBEY files 285 286 purging old trail files regularly 287 reporting regularly on process health 286 316 Manager parameter file modifying 193 195 using built in editor 193 using GUI 194 Manager process 40 193 219 228 Manager status verifying 58 59 MANAGESECONDARYTRUNCATIONPOI NT 62 Manual Mapping tab Compare Pair Mapping screen 178 MAP parameter configuring for initial load Replicat file 73 74 for Replicat process 80 81 MAP statements 96 110 222 239 mapping advanced 210 212 columns 97 98 rules 197 marker_setup sql script 103 MAX option 223 MAXDISCARDRECS parameter 224 maximum threshold 155 MAXTRANSOPS parameter 238 MEGABYTES option DISCARDFILE parameter 222 225 www it ebooks info memory requirements Oracle GoldenGate 16 Oracle GoldenGate Veridata 28 29 scripts monitoring 168 MGRPORT parameter 231 Microsoft SQL Server 2000 112 Microsoft SQL Server 2005 112 Microsoft SQL Server 2008 112 Microsoft SQL Server databases to Oracle replication 111 118 database configuration 113 118 initial data load c
324. l Extract has already captured You can configure a single data pump or multiple data pumps depending on the requirements For example a data pump on the source system can pump data to an intermediate or middle tier system The data on the middle tier can be further filtered by multiple pumps running in parallel and passed on to multiple heterogeneous targets In this case no database is required on the middle tier only the data pumps Figure 3 3 demonstrates multiple data pumps running in parallel Data Pump 1 and Data Pump 2 can pump data on to another pump or to one or more Replicats 37 www it ebooks info CHAPTER ARCHITECTURE 38 esses Capture Source gt Local Data Pump D Data Pump D Database 1 Extract J i ip p Source Trail E E Figure 3 3 GoldenGate flow with multiple data pumps Just as with the Local Extract you can write any records the data pump is unable to process to a discard file for problem resolution Reports can be automatically generated to show the data pump configuration and updated periodically at user defined intervals with the latest processing statistics Network GoldenGate sends data from the source trail using the Local or data pump Extract over a TCP IP network to a remote host and writes it to the remote trail file The Local or data pump Extract communicates with another operating system background Extract process called the Collector on the target Th
325. l and configure the following systems e Oracle GoldenGate Veridata Web e Oracle GoldenGate Veridata Server 27 www it ebooks info CHAPTER 2 INSTALLATION e Oracle GoldenGate Veridata Agent e Oracle GoldenGate Veridata CLI e Source and Target databases to be compared We will cover administration of these Oracle GoldenGate Veridata components in Chapter 9 For now we will provide you with details on the installation process First we will discuss the requirements for the Oracle GoldenGate Veridata Agent GoldenGate Veridata Agent System Requirements You will need to install an Oracle GoldenGate Veridata agent for each database instance to be compared in the comparison check In addition the following software is required for the Oracle GoldenGate Veridata Agent e Java is required for the agent to function Install either the Java Runtime Environment e JRE 1 5 Java 6 or the Java Software Developer Kit SDK for UNIX and LINUX systems that will be used with the Veridata Agent Note A TCP IP network port must be available and configured for all database platforms that will interact with Veridata Agent and Server For instance with Oracle you must have the Oracle listener configured and online GoldenGate Veridata Agent Disk Requirements While there is no exact disk space requirement for the Oracle GoldenGate Veridata Agent we recommend that at least 200MB of disk space be granted for the software installation
326. l load process Configuring the Replicat for Disaster Recovery After the initial data load you can start the Replicat to apply the changes that have been captured by the Extract while the load was running If needed turn on the HANDLECOLLISIONS parameter to resolve any errors with missing or duplicate data It will take some time for the Replicat to catch up with applying all the changes made during the load particularly for a large database with heavy change volume and a long load time Once the initial data load changes have been applied by the Replicat and there is no GoldenGate lag left the primary and standby databases will be fully synchronized At this point the Local and data pump Extracts and Replicat can continue to run and keep the databases synchronized in real time with the ongoing changes Before starting to configure the Replicat you should go back and double check the GoldenGate prerequisites have been met on the target standby server We covered those at the beginning of the chapter in the section Prerequisites Once you have confirmed the prerequisites are met let s begin configuring the Replicat Remember you re configuring the Replicat to apply all the DML changes from the HR schema based on our disaster recovery requirements to replicate all the data If you wanted to filter or transform data you could also do that here by adding the appropriate parameters Since you already used RHREMPI for a Replicat name earlier you
327. l mode 2011 03 22 17 12 15 WARNING 0GG 01003 Repositioning to rba 231493300 in seqno 49 When these errors occur GoldenGate automatically rolls back the BATCHSOL processing and attempts to process the transaction in normal mode This slows down the BATCHSQL so if you re seeing a lot of these errors you may find BATCHSQL won t improve your performance When you have BATCHSOL successfully enabled you should review your Replicat report file and compare the processing rates to your baseline reports and determine if you re getting performance improvements Using GROUPTRANSOPS You can use GROUPTRANSOPS to tune the number of SQL operations in a GoldenGate transaction As you saw in Figure 7 5 the GoldenGate Replicat groups together source transactions into a GoldenGate transaction By default the Replicat includes a minimum of 1 000 SQL operations in a transaction By setting GROUPTRANSOPS higher say to 2 000 you may get increased performance In addition to having a larger transaction a higher value for GROUPTRANSOPS reduces the number of checkpoints written to the GoldenGate checkpoint table GROUPTRANSOPS can be set in your Replicat parameter file as shown in the following example GROUPTRANSOPS 2000 Be careful not to set GROUPTRANSOPS too high Doing so can cause an increase in latency on the target because the target transaction size is larger on the target than the source Tuning Disk Storage By its very nature as a data replicatio
328. ld give a similar number of operations if there is no special filter in the mappings On Source GGSCI stats dpggsi totalsonly GGSCI stats extggs1 totalsonly On Target GGSCI stats repggsi totalsonly If you see different numbers of operations then there may be data lost or filtered in the process For example in one way replication all the stats from the Extract and Replicate side should match If you have 10 records inserted into the source table but only 9 in target there must be something wrong Monitoring Lags for Each Group There are different ways to find out the lags in the GoldenGate process All the approaches are similar and you can use one or all of them to validate the lags Using getlag The first approach uses the getlag command GGSCI lag extract extggs1 Or GGSCI send extract extggs1 getlag Sending GETLAG request to EXTRACT EXTGGS1 Last record lag 1 seconds You get one of the three statuses e At EOF no more records to process There is no lag in the Extract data pump or Replicat If the data still isn t replicating check the trail file names or the database connection to make sure they re set up properly e Waiting for data to process In the Extract Extract process is waiting for the redo log In the data pump Extract is waiting for the Extract trail files In the Replicat Replicate process is waiting for the Data Pump or Extract trail files The lag may be the network
329. ldenGate that complain about a missing library you should issue the env command from a shell window to check the settings for the LD LIBRARY PATH and PATH variables to ensure that they re correctly set If these variables are incorrect rectify the situation and set these values to the correct path in the profile startup file in the Oracle GoldenGate home directory The error report may also display a message that the function stack needs to be increased To increase the memory allocated for the stack you can use the FUNCTIONSTACKSIZE parameter www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE Note Be careful with the FUNCTIONSTACKSIZE parameter because it may adversely affect Oracle GoldenGate performance Be sure to first test it in a nonproduction environment when you make changes File access problems can also cause failures at the operating system level Both Extract and Replicat user accounts at the operating system level required full read and write permissions for all files in the Oracle GoldenGate directory If you receive the error message Bad parameter Group name invalid it indicates that the process can t open the checkpoint file You should execute the GGSCI command INFO to view the group name and then issue the command VIEW PARAMS group to make sure the group name from the GGSCI output for the INFO command matches the one in the EXTRACT or REPLICAT parameter Finally check the values for the ke
330. ldenGate touches multiple areas including database network storage and operating systems the chapter in turn addresses these areas as related to Oracle GoldenGate problem resolution In preparation for this discussion it s useful to take a visual approach to Oracle GoldenGate root cause analysis as shown in Figure 11 1 www it ebooks info 217 CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 218 Network Operating System Database Application Server Storage Disk Hardware Figure 11 1 Holistic approach to troubleshooting Oracle Goldengate software This chapter examines the following areas that can cause grief in Oracle GoldenGate environments e Process failures e Trail file issues e Synchronization problems e Startup problems on source and target systems e Database configuration and availability issues As part of your journey into troubleshooting Oracle GoldenGate this chapter provides tips and techniques you can use to trace and identify the source of problems so you can quickly solve them Let s get started Oracle GoldenGate Process Failures The first type of problem you may encounter within Oracle GoldenGate environments are process failures These may occur on the source on the target or on both environments for one or more of the Oracle GoldenGate processes Recall from earlier chapters that an Oracle GoldenGate environment consists of many different background processes on the source and target
331. le up to larger environments and move to Oracle environments as shown in Table 6 2 Table 6 2 Supported Configurations for Oracle 11g GoldenGate with MySQL Source Source OS Target Target OS GG Version MySQL Microsoft Windows Linux Oracle 91 10g Windows Linux Oracle 11g 5 UNIX 115 UNIX GoldenGate Details on how to configure MySQL to Oracle replication can be found in the online Oracle GoldenGate documentation guides available at http otn oracle com 122 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION Teradata to Oracle Replication Teradata is one of the key players in the enterprise data warehouse and business intelligence market It scales well and has robust performance However lack of a built in native replication feature has been the Achille s Heel of Teradata As such it requires Oracle GoldenGate software to perform database replication tasks Many customers have a business need to replicate from Teradata to Oracle environments and GoldenGate fulfills this task perfectly Table 6 3 provides the currently supported configuration of Teradata with Oracle GoldenGate Table 6 3 Supported Configurations for Oracle 11g GoldenGate with Teradata Source Source OS Target Target OS GG Version Teradata Microsoft Windows Linux Oracle 9i 10g Windows Linux Oracle 11g 12 0 UNIX 115 UNIX GoldenGate Teradata 13 0 Details on how to configure Teradata to Oracle replication can be found in the
332. lete Constraints You need to disable any database triggers or cascade delete referential integrity constraints on your targettables The reason is to prevent duplicate changes because GoldenGate is already replicating the results of the trigger and cascade delete operations from the source database If you didn t disable the constraints and triggers GoldenGate would replicate those changes then the triggers and cascade delete constraints would also fire causing duplicate changes on the target database Several methods are available for disabling the triggers and cascade delete constraints The first method is only available for newer Oracle versions and involves adding parameters to the Replicat The second method is simply to alter the tables to disable the triggers and constraints and then enable them when replication is turned off Starting with GoldenGate version 11 a new SUPPRESSTRIGGERS option is available as part of the Replicat DBOPTIONS parameter to automatically suppress the triggers from firing on the target You can use it to avoid having to manually disable the triggers This option is available for Oracle 10 2 0 5 databases and later and for Oracle 11 2 0 2 databases and later To disable constraints if you have Oracle 9 2 0 7 database and later you can use the Replicat parameter DBOPTIONS with the DEFERREFCONST option to delay checking and enforcement of integrity constraints until the Replicat transaction commits You can also u
333. lication In this chapter you ll build on that basic configuration and implement a specific replication configuration for disaster recovery purposes In many ways the GoldenGate configuration for disaster recovery is similar to basic replication with a few enhancements This chapter will cover setting up and configuring GoldenGate replication for disaster recovery We ll cover how to handle both planned and unplanned outage scenarios and how you can use GoldenGate to minimize downtime Prerequisites You need to have the following prerequisites in place before you can start setting up replication for disaster recovery e The GoldenGate software installed on the source and target servers as described in Chapter 2 e The GoldenGate database User ID created on the source and target databases server name or IP address of the target database server e The GoldenGate Manager process up and running on the source and the target e TCP IP network connectivity open from the source server to the target server s GoldenGate manager port and vice versa e Anunderstanding of the business and technical replication requirements for the disaster recovery replication 241 www it ebooks info 12 242 DISASTER RECOVERY REPLICATION There are a few additional prerequisites that are important for disaster recovery replication Backups of your GoldenGate software files and working directories on the source and target servers
334. log shown in Figure 10 5 opens It s a very nice feature that this dialog shows all the available Manager parameters and attributes along with their online help www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Insert Parameters Value c ggs dirdativa Attributes JE Min Keep Hours o Min Keep Days o Min Keep Files o E PURGEOLDEXTRACTS Use PURGEOLDEXTRACTS to delete old trail files when processes have finished with them or to eliminate old data from the GoldenGste log table Preventing the accumulation of data conserves disk space Using PURGEOLDEXTRACTS Manager parameter is preferred over using it ss Capture or Delivery parameter because it allows you to manage the files in a centralized fashion You can i Figure 10 5 Inserting a parameter into the Manager parameter file Select purgeoldextracts from the list at left and enter your dirdat full path with a two character trail file prefix Then click Insert You can add other parameters if you wish or click the Done button if you re finished Right click the data source and choose Restart to refresh the Manager You can also go to GGSCI and enter refresh mgr Next you walk through several tasks using Director Client Setting Up the Initial Load The Oracle GoldenGate Initial Load task is very easy to set up in Director It has a menu item to automate this you only need to map the table s
335. lowing commands on the source Oracle database system to create the DEFGEN parameter file and to add the additional parameters given below cd ggs ggsci GGSCI gt edit param defgen DEFSFILE dirdef source def PURGE USERID GGS PASSWORD xxxx TABLE GGS EMP TABLE GGS DEPT GGSCI gt exit Execute the Source Definition Generator on the Oracle Source System Execute the defgen commands on the source Oracle database system defgen paramfile dirprm defgen prm Transfer the Source Definition File to the Target Microsoft SQL Server System ftp mssql2k8 Password ftp gt ascii ftp cd ggs dirdef ftp lcd ggs dirdef Now that we have configured the Oracle source database system for Oracle GoldenGate we will discuss how to set up the target environment to replicate data from Oracle to Microsoft SQL Server 114 www it ebooks info CHAPTER 6 M HETEROGENEOUS REPLICATION Target Microsoft SQL Server Database Configuration Before you can replicate data from the source Oracle database environment there are a number of preliminary setups that must be performed Let s walk through each of these procedures 1 Configure ODBC source database connections Set up permissions and security in source database Create Extract and data pump processes and parameter files gt Enable supplemental logging 5 Add manager to Windows as a service optional Install the Oracle 11g GoldenGate software for Microso
336. mer fulfliment For palches see My Oracle Supper Figure 2 7 Download Oracle Goldengate from OTN website On the Software Downloads heading on the OTN website you will need to accept the license agreement before downloading Once you do so you can download the Goldengate software for your platform to use on a trial educational and non production basis Be sure to navigate to the heading for Middleware software as shown in Figure 2 8 since Oracle classifies Goldengate as part of the Oracle Fusion Middleware product family Middleware Oracle Fusion Middleware 119 Release 1 Forms amp Reports Services Download page includes WebLogic Server WebCenter Identity Management HTTP Server and other components Foundation Pack Application Server 10g and Components BPEL Process Manager ide Management Imaging and Process Management nformation Rights Management Business Acti onitorin JRockit Business Intelligence Suite EE Outside In Technol Business Intelligence SE Portal Business Intelligence SE One Presence Business Intelligence Applications Business Intelligence Publisher Business Process Analysis Suite Business Process Management Real Time Decisions Service Bus Service Bus for Financial Services Service Registry Figure 2 8 Download Oracle Goldengate from OTN website 11 www it ebooks info CHAPTER 2 INSTALLATION 12 Click the Golden
337. mine the before and after image of the transactions contained in the trail file as displayed by Logdump For additional details on using the Logdump utility with Oracle GoldenGate consult the Oracle GoldenGate Troubleshooting Guide which contains a list of all Logdump commands syntax and usage It s available online at http download oracle com docs cd E15881 01 doc 104 gg troubleshooting v104 pdf for release 10 x and at http download oracle com docs cd E18101 01 doc 1111 e17792 pdf for release 11 1 www it ebooks info Index A Abend failures Replicat process 236 237 access log file 193 active active replication 43 106 129 ADD command 293 ADD EXTRACT command 67 229 236 ADD REPLICAT command 91 ADD RMTTRAIL command 67 121 ADD TRACETABLE command 109 ADD TRANDATA commands 55 56 239 Administrator component Director 190 Administrator role name Tomcat Web Server Administration Tool screen 187 advanced replication 2 3 agents C Code 173 Java 173 Veridata 173 alerts Checkpoint Lag 213 Event Text 213 set up 214 setting up with Director application 212 214 alias Extract 294 ALTARCHIVELOGDEST lt path name gt parameter 154 235 alter extract begin now l timestamp command 156 ALTER EXTRACT command GGSCI 229 ALTER EXTTRAIL command GGSCI 222 ALTER REPLICAT lt group gt EXTTRAIL lt trail file name gt command GGSCI 236 ALTER RMTTRAIL command GGSCI 222 ALTE
338. mp as shown in this example 65 www it ebooks info CHAPTER 4 M BASIC REPLICATION 66 GGSCI sourceserver 1 edit params PHREMD1 Extract PHREMD1 PassThru RmtHost targetserver MgrPort 7840 RmtTrail dirdat 12 Table HR Now let s look at each of the parameters in more detail As mentioned with the Local Extract GoldenGate has many more configuration parameters than are used in the basic example You can refer to the GoldenGate Reference Guide for a detailed description of all the parameters I usually refer to the Alphabetical Reference section in the GoldenGate Reference Guide to quickly find a parameter and its meaning and various options EXTRACT You probably already noticed that the data pump is simply another type of GoldenGate Extract Use the Extract parameter to define the GoldenGate data pump Extract group name Just as with the Local Extract you re limited to eight characters for Extract group name Follow the same naming standard covered in the previous Extract section The example named the data pump Extract group PHREMDI The P specifies that it s the data pump Extract The rest of the name is the same as the Local Extract covered in the previous section You can add additional parallel data pump Extract groups to improve performance as discussed more in Chapter 7 Tip You don t specify a SETENV in the data pump SETENV for NLS LANG isn t needed for the data pump Extract because it has no effect
339. n Chapter 5 You can also specify NODBCHECKPOINT to indicate that the Replicat will only write checkpoints to a file on disk After adding the Replicat you need to start it to actually begin applying changes to the target database The following example starts the Replicat RPREMPI on the new server GGSCI newserver START REPLICAT RPREMP1 You can verify that the Replicat is running using the INFO REPLICAT command You should see a Status of RUNNING If you see a Status of STOPPED or ABENDED there may be a problem Let s do an INFO command on the Replicat RPREMPI to check the status GGSCI newserver 2 INFO REPLICAT RPREMP1 After verifying that the Replicat is running you should also confirm that the Replicat is processing data using the STATS command Note When you re finished synchronizing the old and the new databases you should perform data validation There may be cases where the data isn t replicated properly or perhaps there is an error in the configuration You can do data validation manually with SQL queries for row counts and some selected data values or use an automated tool such as Oracle GoldenGate Veridata Refer to Chapter 9 for more information on Veridata Next let s add the Oracle GoldenGate processes for the replication from the new database back to the old database 271 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION 272 Configuring the Fallback Local Extract for Zero Do
340. n jobs are both CPU and I O intensive Veridata Server performs the following functions e Coordinates all aspect of the Oracle GoldenGate Veridata execution tasks with the agents e Sorts rows on Veridata Server The default is to sort on the database that the agent is installed on e Compares the data collected from the GoldenGate Veridata Agents e Conducts the confirmation comparisons of out of sync data to be explained later e Produces out of sync confirmation and performance reports 172 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA GoldenGate Veridata Web Oracle GoldenGate Web is a web based client to the Veridata server This is the tool that we will be using to configure most of the Veridata tasks It performs the following functions e Configures all comparison objects and rules including connections groups jobs etc e Starts and stops the comparison jobs e Provides real time status on the job status e Reviews the comparisons and out of sync data reports GoldenGate Veridata Repository Veridata Repository stores all the configurations for all the comparison objects and rules The metadata is stored as clear text in the database tables in the repository database so you can query them to create your own custom reports It also stores the job status so you can integrate it with your ETL and monitoring tools seamlessly GoldenGate Veridata Agent Java and C Code Veridata Agent executes t
341. n petabyte environments As such companies today look to migrating off of their smaller Microsoft SQL Server environments to Oracle for the robust performance that Oracle can provide You can verify the currently supported third party database platforms that Oracle 11g GoldenGate supports 111 www it ebooks info CHAPTER 6 HETEROGENEOUS REPLICATION by logging into the My Oracle Support MOS site online at http support oracle com with a valid customer account CSI number and checking the certification matrix as shown in Figure 6 1 DRACLE ORACLE SUPPORT Welcome Ben Contact Us Dashboard Knowledge Service Requests Patches amp Updates Community Certifications Systems Collector Reports More v Search Knowledge Base Certifications Last New Certification Search We ve improved how you can search navigate and understand certifications Find certifications in 3 easy steps O Find your product Pick a product release G Click Search Use the auto suggest field to easily find your product Select the release of your chosen product in the dropdown list Find all certified products and platforms Product Release a 11 2 0 1 0 Database Vault DB Vault Oracle Database RDBMS 106 3i DB Oracle Server Enl 11 10 70 Oracle Database RAC RDBMS Real Application Cluster 1110560 Quick Links Certification Search Ls Search Saved Recent c 4
342. n product GoldenGate generates a lot of disk I O Trails are written by the Local Extract read and written by the data pump Extract and finally read by the Replicat As such to achieve the highest performance it s critical to place your trail files on the fastest possible disk storage Here are some suggestions for tuning disk storage for GoldenGate to achieve the best performance 149 www it ebooks info CHAPTER 7 88 TUNING e Place GoldenGate files on the fastest available storage and storage controller Although NAS is acceptable for GoldenGate files generally SAN storage has better performance than NAS due to the NAS network overhead Storage technology is constantly changing and performance should be evaluated at the time of your decision e Use RAID 0 1 rather than RAID 5 RAID 5 maintains parity information that isn t needed and slows down writing the trails e Always evaluate the performance of the disks used to store your trail files For example on Linux use the IOSTAT command to determine if any disk devices are a bottleneck e Use the GROUPTRANSOPS parameter as described in the last section to reduce I O Tuning the Network Depending on the number of data changes in your replication environment the Extract could be sending large amounts of data over the network to be processed by the Replicat Network performance between your source and target is critical to overall GoldenGate performance As mentioned earlier you
343. n screen as shown in Figure 9 3 It will take you to the Compare Pair Mapping screen as shown in Figure 9 4 Existing Compare Pattern Mapping Manual Mapping Preview Datasource Information Source Target Schema VERIDATA1 TE Schema VERIDATA2 IM Manual Compare Pair Mapping Source Tables Totak 2 Target Tables Totak 2 Filters Filter Applied Filters Filter Applied Select Mapped Table Name Select Mapped Table Name BIG TABLE1 I7 TABLE1 c i7 TABLE SOURCE f 7 BIG TABLE TARGET M 4 gt M M 4 gt M Show 10 items per list Compare Pair Details Compare Pair Name BG TABLE SOURCE BIG TABLE TARGET Profi z Row Partitions Not Specified Key Mapping Method System Generated Figure 9 4 Compare Pair Mapping In the Compare Pair Mapping screen you can use Manual Mapping to add the compare pairs After you pick up the compare pair make sure your compare name is in the Compare Pair Detail section In the example shown BIG_TABLE_SOURCE BIG_TABLE_TARGET is the compare pair name Click the Preview tab to confirm the settings Creating a Job Jobs are like shell scripts they are used to run a series of compare groups There is no built in scheduler in Veridata so you use the OS schedulers crontab Windows schedulers torun vericom command line jobs A job name and a group are the only required parameters You can check the group name big_group_table that you just created in
344. n t recorded in the Oracle database archive logs It can take hours or even days for the Extract to search through the Oracle database archive logs to locate the requested log file If the Extract fails to locate the archive log it abends and fails You should run a query against the V TRANSACTION dynamic performance view on the source Oracle database to ensure that no open transactions are present You should also run the GGSCI SEND EXTRACT command on a regular basis to view and manage long running transactions One way to set a threshold for long running Oracle database transactions is to use the parameter WARNLOGTRANS to identify long running transactions in Oracle GoldenGate In the event that your Oracle database online redo logs have dumped their current data to the archive logs and you can t restore the necessary archive logs you must resynchronize both the source and target database environments Extract Failure Conditions Due to Oracle Source Database Issues As mentioned earlier Extracts must have read access to log in to the source Oracle database to read data from the online redo log files Check the permissions for the Extract user account provided in the parameter file if you experience errors during operation You can verify this by using the DBLOGIN command to check for permission and read access issues Another root cause of this issue is the disk full condition for the source Oracle database file system Once you ve resolved the rea
345. n t specify target tables or column mapping e When configuring a mapping statement for a Delivery object specify the source and target tables If needed map source columns to target columns and specify column conversion functions On the diagram click Map for EXTINIT and then click the Insert button at the bottom The Insert New Mapping dialog opens For now use the Mapping Type value Table as shown in the Figure 10 8 Enter the table name in the Table Name text box Click Insert for each new mapping 197 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Insert New Mapping TABLE is a synonym for the parameter One exception for TABLE the TARGET parameter is optional Figure 10 8 Insert New Mapping dialog for the Extract Your mapped tables are checked on the first column of the Parameters tab as shown in Figure 10 9 If you don t see them click the Show Tables button You also can select tables in the table list panel as shown in the Figure 10 9 198 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Figure 10 9 Mapping table list for the Extract To view or edit your Extract parameter file right click Extract for EXTINIT and choose Advanced Edit Param The dialog shown in Figure 10 10 opens Param File Editor extract EXTINIT userid gg director2 password gg director2 discardfile C ggs dirr
346. nGate mappings using Director Client on the desktop You skip some Oracle GoldenGate concepts because they re covered in previous chapters the Administrator s Guide the Reference Guides and Director Online Help Director Client is a GUI application This chapter includes a number of screenshots so you can be sure you re on the same page as you follow along with the examples In Director Client the same menu bar or button may behave differently depending on which object you ve highlighted For example the View Detail menu item displays different dialogs when you highlight Extract Object Data Source and Trail Files It s important to follow the steps at the beginning of the chapter you skip some basic instructions later When you re familiar with the Director interfaces some tasks are easier and faster than the GGSCI alone With Oracle GoldenGate Manager running you don t even need to touch vi or Notepad to perform most common development tasks Setting Up the Data Source Before you use the Director application you have to set up the data source using Oracle Golden Gate Director Admin Tool Make sure Director Server is already running and then follow these steps 1 Enter the Username and Password as shown in Figure 10 3 The default is admin for both Username and Password toon 5 ORACLE GoldenGate Director User Name e Password Server locaihost 7001 cm Figure 10 3 Director login screen www it ebooks
347. navigation tree to the local SQL Server Instance Right click the Microsoft SQL Server database instance as shown in Figure 6 5 and select Properties Console Root Microsoft Console Root Microsoft SQL Servers J SQL Server Group SBE local Windows NT Databases 09554 995592 9 master BJ model msdb 9 Northwind pubs 9 tempdb 0 8 Figure 6 5 Creating the sample Microsoft SQL Server database 118 www it ebooks info CHAPTER 6 88 HETEROGENEOUS REPLICATION In the following Microsoft SQL Server Window select the Security tab and then choose authentication for SQL Server and Windows as shown in Figure 6 6 SQL Server Properties Configure local Server Settings Database Settings Replication Active Directory General Memory Processor Security Connections Security E SOL Server provides authentication based on Windows v accounts and a named SOL Server login ID and password Authentication C Windows only Audit level None Failure Success C All Ownership chaining Allow cross database ownership chaining r Startup service account Start and run SQL Server in the following account System account C This account Password Cancel Figure 6 6 Choosing the Security tab for Microsoft SQL Server database Create the sample database on the
348. nd target Change_date between sysdate 1 and sysdate Veridata will compare the data that has changed since yesterday Comparing GoldenGate Real Time Replication Data If there is no timestamp in your table then the confirm out of sync option is the only option to make sure the data is consistent There are three statuses for confirmation e In flight row was out of sync in the initial compares step but it has changed Veridata cannot confirm it is in sync yet e In sync row was out of sync in the initial compares step it is in sync now e Persistently out of sync row was out of sync in the initial compares step but is still out of sync after the confirmation step By default the initial compares and confirmation steps are running in parallel It waits for 60 seconds between two steps This value can be changed in the profile setting Comparing Different Column Types and Compare Formats You can override the default Column Mapping name type and even format In Compare Pair Configuration click the column name hyperlink you will see the dialog in Figure 9 6 You can type over the Source Column name Source Datatype and their compare formats This is useful if source and target have different column names and even different data types although they are actually the same data www it ebooks info CHAPTER 9 88 ORACLE GOLDENGATE VERIDATA ColumnPair Configuration Column Pair Details Source Table BIG_TABLE_SOURCE
349. nd Oracle data warehouse capabilities many IBM customers have a business need to maintain their current DB2 database environments while maintaining their Oracle database environments As a result they have a need for a heterogeneous replication solution to keep all of their database environments in sync Oracle GoldenGate fulfills this key requirement Oracle 11g GoldenGate provides support for the following versions of IBM DB2 UDB with Oracle database environments as shown in Table 6 5 Table 6 5 Supported Configurations for Oracle 11g GoldenGate with IBM DB2 UDB Source Source OS Target Target OS GG Version IBMDB2UDB Microsoft Windows Oracle9i 10g Windows Linux Oracle 11g 8 x Linux UNIX 116 UNIX GoldenGate IBM DB2 UDB Microsoft Windows Oracle 91 10g Windows Linux Oracle 11g 9 x Linux UNIX 116 UNIX GoldenGate Details on how to configure IBM DB2 UDB to Oracle replication can be found in the online Oracle GoldenGate documentation guides available at http download oracle com docs cd E18101 01 doc 1111 e17795 pdf Verifying Operational Readiness Now that we have set up multiple replication environments with third party databases to move data to Oracle it s time to test and verify that the new Oracle GoldenGate environments function correctly We also need to test and measure basic performance to ensure that sufficient resources are available to move data to Oracle from our different source systems Operational read
350. nd operating system Are the servers stand alone or clustered How is the data storage configured What are the character sets for the source and target databases Data currency How quickly and how often does the data need to be updated Can any lag in the data replication be tolerated If no lag can be tolerated and there is a high volume of changes you may need to devote more time in the project to designing and tuning the replication to avoid any lag Keep in mind that often reporting systems can tolerate lag and the target data doesn t need to be completely up to date with the source Data volume How much data needs to be replicated How often is it updated You may be able to examine the existing database transaction log files to determine the amount of data change that is occurring in the source database The data change volume will impact the network bandwidth requirements between the source and target and the amount of disk space required for trail files Data requirements You should understand any unique or special data requirements so they can be handled properly in GoldenGate For example if there are triggers they may need to be disabled in the target database Sequences need special parameters added if they need to be replicated Certain data types may need special handling or may not be supported For example in GoldenGate 11 data types such as BFILE and ANYDATA aren t supported Other data types may be supported but wit
351. ndows Microsoft SQL Server Oracle 11g GoldenGate Target System SQL XP 2008 v11 1 1 0 Server We recommend that you download the Oracle Goldengate 11g software for MySQL and Microsoft SQL Server platforms from either OTN or Oracle E Delivery in preparation for the exercises we have planned in Chapter 6 on heterogeneous replication with Goldengate If you need the MySQL database software you can download it from www mysql com downloads enterprise Microsoft SQL Server 2008 Personal Express Edition database software is available from www microsoft com express Database or alternatively you can use the trial license version of Microsoft SQL Server 2008 Enterprise Edition to perform the exercises in this book You can download a 180 day license for the trial version from www microsoft com sqlserver 2008 en us trial software aspx Now let s take a voyage into the system and database prerequisites that must be performed before installing Oracle Goldengate software Reviewing the Install Instructions The Oracle GoldenGate documentation provides a multitude of installation and configuration guides that are available from OTN at www oracle com technetwork middleware goldengate documentation index html Figure 2 10 shows a list of guides that explain the installation configuration and support procedures for Oracle Goldengate www it ebooks info CHAPTER 2 88 INSTALLATION ORACLE Fusion Middleware 11g Release 1 11 1 1 WELCOME TO T
352. ndows or shell scripts in UNIX that you OBEY files in the diroby directory under your GoldenGate installation You can name the OBEY files anything but it s a good idea to have a naming standard such as using a standard filename extension like obey Let s look at an example OBEY file Assume you want to add an Extract for your production payroll data named LPAYRPI along with the corresponding trail file You can create an OBEY file to add the Extract as shown in the following example cat diroby LPAYRP1 obey ADD EXTRACT LPAYRP1 TRANLOG BEGIN NOW ADD EXTTRAIL gger dirdat 11 EXTRACT LPAYRP1 MEGABYTES 25 GGSCI sourceserver 1 obey dirdat LPAYRP1 obey 285 www it ebooks info CHAPTER 14 TIPS AND TRICKS 286 Using OBEY files gives the added advantage that your files are stored centrally as a good reference and can be reused You can create a standard set of OBEY files and ensure a consistent reusable scripted process for your GoldenGate setup and maintenance Anyone supporting the replication can easily find the files you used to set up the replication In addition if something goes wrong you always have your scripts to use as a reference point Generating Interim Statistics By default GoldenGate generates a report each time an Extract or Replicat process is started In addition you should include the REPORT parameter to generate interim runtime statistics at least on a daily basis This information can be inval
353. nnecting to the Database For Oracle RAC you should set up a database service specifically for the GoldenGate connection and define the connection information in your tnsnames ora file You can set up the service using the Oracle Database Configuration Assistant DBCA or with Oracle Enterprise Manager as shown in Figure 5 1 100 www it ebooks info CHAPTER 5 ADVANCED FEATURES Cluster Managed Database Service GoldenGate The service has been configured to run on the following instances A service may have been stopped on an instance if the instance was down or the service was disabled Starting a service on a down instance will first bring up the down instance Senice Status v Service is running on all preferred instances CPU Load v Transparent Application Failover TAF Policy NONE Top Consumers Details Service Properties Edit Instances Enable Disable Start Stop Relocate Select Instance Name Service Status for Instance Instance Status Service Policy Response Time per user call milliseconds CPU Time per user call milliseconds Status Details sourcedbi 4 Running t Preferred 0 00 v 0 00 sourcedb2 Stopped t Available n a n a v Figure 5 1 GoldenGate service defined in OEM The example in Figure 5 1 defines the service on the preferred instance sourcedb1 and it s available on instance sourcedb2 After the service is defined you can stop the Extract or Replicat and move the service from i
354. nnection refused it indicates that the target Manager or Server isn t running or that the Extract process is pointing to the wrong TCP IP address or Manager port number The report for the Extract shows an error in GGSCI similar to this ERROR sending message to EXTRACT EATAA TCP IP error Connection reset You can use the GGSCI command INFO MGR to identify the port number in use by the target Manager Another item to check the Extract parameter RMTHOST to ensure that MGRPORT is using the same port number as shown in the GGSCI INFO MGR command If you used a host name be sure to check that the server s domain name server DNS can resolve it If an IP address was used make sure to verify that it s correct You can issue the IFCONFIG command for Linux or UNIX and the Windows IPCONFIG command to verify an IP address from the OS command shell In addition you should test network connectivity 231 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 232 between the source and target systems by using the PING host name command You can display the network routing table by using the NETSTAT command to check for routing access between source and target Another common network error that occurs with Oracle GoldenGate appears when the Extract returns the error No Dynamic ports available This means the target Manager process was unable to obtain a port on which to communicate with the source Manager The Manager process lo
355. nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn munanna 33 Typical mr P 33 GoldenGate Components 34 Source Database H 34 Capture Local Extract PFOC8SS etuer en ttn onaniaa anaana anaana nett ao oa a oa Eno KE E RARE RR RR 35 V www it ebooks info CONTENTS Trail 36 PUMP ei 36 NOeUWOIK sssaininisshiciusacssntnsesbssindnatontaisctuodasashandagasbiainiestsatadsensndesssuraiusastandasudbantedeatiadasudnbon Du 38 iuumm 38 Remote Trail III 38 wa mii 39 Target Database 39 U 40 Topologies and Use MET 40 USEACUU 40 Bidirectional Replication eee tnter tnnt tentent tantes 43 Broadcast 44 Integration Replication ener rennen 45 1001S X 46 Hu E 46 uil TEE 46 LOQQUIMD 46 46 46 47 47 Chapter 4 Basic Replication
356. nology with Oracle database systems Before the chapter discusses the foundations of Oracle GoldenGate software a brief history of time is warranted for how database replication technologies came into existence Before GoldenGate software data transactions were replicated across networks with simple file transfers via the File Transfer Protocol FTP for nonrelational databases With the popularity of the UNIX operating system and the advent of client server computing data replication was implemented with Oracle basic and advanced replication software in version 8 of Oracle database software Distributed Processing and Replication Oracle release 5 introduced distributed processing queries in the form of database links Database links provided the ability to issue SQL queries across networks and were the first real attempt to achieve a replication solution Figure 1 1 shows how connectivity is established between the source and target database environments via database links www it ebooks info CHAPTER 1 88 INTRODUCTION SCOTT EMP Figure 1 1 Database links for distributed systems The next step in the march of data replication came into existence with Oracle release 8 which enabled database professionals to set up log based and trigger based replication solutions Oracle Basic Replication Oracle basic replication existed in two flavors log based and trigger based With log based repli
357. nsaction consistency between heterogeneous environments without the need to develop complex routines or code Summary This introduction has provided a history lesson on how database replication technologies evolved into the robust Oracle GoldenGate technologies The next chapter shows you how to install the Oracle GoldenGate product suite www it ebooks info 2 Installation The installation process for Oracle GoldenGate is simple compared to most of the other product suites available from the Oracle Fusion Middleware family of which GoldenGate is a part In this chapter we will provide you with details on how to prepare for and perform a complete installation of the following Oracle GoldenGate products e Oracle GoldenGate 11g e Oracle GoldenGate Director 11g e Oracle GoldenGate Veridata 11g Downloading the Software For Oracle GoldenGate 11g the first step is to obtain the software either online or from DVD media Depending on the bandwidth of your Internet connection we recommend that current Oracle customers download the Oracle GoldenGate software from Oracle E Delivery at http edelivery oracle com For educational and non business learning purposes Oracle provides the software free for download from the Oracle Technology Network OTN at http otn oracle com In this chapter we will download the Oracle GoldenGate software from http edelivery oracle com We recommend that you register for a free accoun
358. nstance winiisrc status READY has 1 handler s for this service The command completed successfully C ggs_src gt tnsping winiisrc TNS Ping Utility for 32 bit Windows Version 11 2 0 1 0 Production on 14 MAR 2 227 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 228 Used parameter files C winora11g2 product 11 2 0 dbhome_1 network admin sqlnet ora Used TNSNAMES adapter to resolve the alias Attempting to contact DESCRIPTION ADDRESS PROTOCOL TCP HOST oracledb a PORT 1521 CONNECT DATA SERVER DEDICATED SERVICE NAME winiisrc OK 80 msec On quick way to verify that the database is online is to use the UNIX or Linux command ps ef grep smon or even better to log on to Oracle SQL PLUS as the Oracle GoldenGate user for the Extract and Replicat as shown here C ggs_src gt sqlplus ggs ggsQwiniisrc SOL Plus Release 11 2 0 1 0 Production on Mon Mar 14 01 59 23 2011 Copyright c 1982 2010 Oracle All rights reserved Connected to Oracle Database 11g Enterprise Edition Release 11 2 0 1 0 Production With the Partitioning OLAP Data Mining and Real Application Testing options SQL gt select count from hr employees COUNT You also need to check to verify that the Manager process is online and running as well as perform the previous checks for the source and target databases Without access to the database systems and the Manager process the Extract and Replicat will fail
359. nstance to instance as needed and then restart the Extract or Replicat You can also display information about the service using the srvct1 utility srvctl config service d sourcedbi s goldengate a goldengate PREF sourcedbi AVAIL sourcedb2 TAF NONE When you ve defined the service you need to add the connection information to your tnsnames ora file as shown here GoldenGate DESCRIPTION ADDRESS PROTOCOL TCP HOST sourceserveri vip PORT 1521 ADDRESS PROTOCOL TCP HOST sourceserver2 vip PORT 1521 LOAD BALANCE NO CONNECT DATA SERVER DEDICATED SERVICE NAME GoldenGate Then you can configure your Extract or Replicat to use the new service based connection as follows USERID GGER GoldenGate PASSWORD userpw This connection connects the Extract or Replicat to the GoldenGate TNS alias you just added which uses the service you defined in the database Now let s look at another consideration you should be aware of when running GoldenGate on Oracle RAC Defining Threads In an Oracle stand alone database there is typically only one redo thread In Oracle RAC there is typically one redo thread per instance You need to tell the GoldenGate Extract process the number of redo threads defined in the Oracle RAC environment The following example adds an Extract in an Oracle RAC environment with two instances and two threads GGSCI sourceserver ADD EXTRACT LHREMD1 TRANLOG THREA
360. ntnnntnnns 76 Steps for Loading with DBMS Utilities eene 76 Starting the ee E 76 Configuring the 3 77 Adding the TTE 81 vii www it ebooks info CONTENTS viii Starting and Stopping the Replicat esesessseseseeseee en enentn nnne tn tne tntne nans 81 Verifying the Replicat rennen nnne nnn nnn natnra 82 S Ir Me 83 Chapter 5 Advanced Features ener nrr nennen nnn nuu uan 85 Enhancing the Replication Configuration enses 85 Enhancing Extract and Replicat Reporting 86 Reporting Discarded Records tu tu tu tu tu t t it t t i enna 89 Purging Old Trail Files eee tnter entente tente tantus 89 Adding Automatic Process Startup and Restart sse 90 Adding a Checkpoint 90 Making the Replication More Secure crecer 92 aevi tica M 92 Encry
361. nts and complexities Determining the topology as soon as possible can help drive the technical requirements and design For example a one way replication topology is much simpler to set up than bidirectional replication Replication involving a single source and target database is easier to set up than a single source and multiple target databases Installation and Setup Now let s look at some tips and tricks related to GoldenGate installation and setup These include suggestions for setting up users specifying Extract and Replicat parameters and naming standards Creating Dedicated Users You may be tempted to simply use the existing database and operating system users for GoldenGate Instead you should create dedicated operating system and database users only for use with GoldenGate Doing so ensures that any activity from GoldenGate users can be tracked separately If there is a database performance issue for example you can isolate activity from the GoldenGate database user to determine if it s causing a problem Some example user names are gger gguser ggate and so on Depending on your platform GoldenGate users may require some elevated privileges that often are reserved for the DBA team For example with Microsoft SQL Server a user with the System Administrators role is required for the Extract process You need to determine based on your organization s security requirements whether the GoldenGate user should belong to the DB
362. o meet the zero downtime migration requirements covered in the last section you have to ensure that you can quickly and accurately switch the application activity from the old database to the new database and back again Also you must be able to replicate back from the new database to the old database in case a fallback is needed from the migration due to problems Figure 13 1 indicates that with a dotted arrow 265 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION 266 Migrate Old New Database 2 Database Fallback Figure 13 1 Zero downtime migration replication topology Now let s look at how to implement the replication requirements and topology using Oracle GoldenGate Setup Once the prerequisites are complete setting up zero downtime migration replication can be accomplished in two parts and seven total steps as shown in Figured 13 2 and 13 3 The steps are similar to setting up basic one way replication as covered in Chapter 4 Part 1 includes the first four steps for setting up replication from the old database to the new database These steps include initially loading the data and then keeping the data synchronized after the initial load Part 2 includes the last three steps for configuring replication back from the new database to the old should it be needed for fallback Because Chapter 4 covered the Oracle GoldenGate parameter files this section only explains new concepts as needed
363. o obtain a listing of the current Extract file locations for additional analysis GGSCI oracledba 26 info extora detail EXTRACT EXTORA Initialized 2011 03 13 23 45 Status STOPPED Checkpoint Lag 00 00 00 updated 00 22 24 ago Log Read Checkpoint Oracle Redo Logs 2011 03 13 23 45 30 Seqno 0 RBA 0 Target Extract Trails Remote Trail Name Seqno RBA Max MB AA 0 0 10 220 www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE Extract Source Begin End Not Available Initialized 2011 03 13 23 45 Current directory C ggs src Report file C Nggs src dirrpt EXTORA rpt Parameter file C ggs_src dirprm EXTORA prm Checkpoint file C ggs_src dirchk EXTORA cpe Process file C ggs_src dirpcs EXTORA pce Error log C ggs_src ggserr log Next let s look at more difficult situations that occur when an Oracle GoldenGate process fails without providing you with a report Oracle GoldenGate Process Failures Without Report Diagnostics If an Oracle GoldenGate process fails before first writing out a report file to standard screen output you can run it from the operating system s command shell not GGSCD to send process information to the terminal To investigate you can issue the startup by using the syntax process paramfile path name gt prm at the operating system level where lt process gt indicates either the Extract or Replicat process for Oracle GoldenGate and paramfile is the fully qualified file name of the parameter
364. og file that contains details for all aspects of replication operations on both source and target systems If an error occurs during GoldenGate processing it s imperative that you review the Oracle GoldenGate error logs located under the source and target base installation directories On Linux and UNIX systems you can use the tail command to examine the last few error messages contained in the error log file for Oracle GoldenGate Let s look at the error log file for the Oracle 11g GoldenGate and Oracle 11g Windows source system The file is called ggserr log you can open it with a text editor such as vi or Emacs on Linux and UNIX or Notepad on Windows platforms In addition you can view the error log file with the GGSCI command VIEW GGSEVT as shown here Oracle PROCESS ABENDING 2011 03 14 00 19 46 INFO 0GG 00992 Oracle GoldenGate Capture for Oracle extora prm EXTRACT starting 2011 03 14 00 19 46 INFO 0GG 01017 Oracle GoldenGate Capture for Oracle extora prm Wildcard resolution set to IMMEDIATE because SOURCEISTABLE is used 2011 03 14 00 19 46 ERROR 0GG 00664 Oracle GoldenGate Capture for Oracle extora prm OCI Error beginning session status 1017 01017 invalid username password logon denied 2011 03 14 00 19 46 ERROR 0GG 01668 Oracle GoldenGate Capture for Oracle extora prm PROCESS ABENDING Understanding the Oracle GoldenGate Discard File Recall from earlier chapters that Oracle GoldenGate creates a discard file wh
365. oks for a port in the list specified with the DYNAMICPORTLIST parameter However if the DYNAMICPORTLIST parameter isn t specified Manager then looks for the next available port higher than the one on which it s running One issue when using the DYNAMICPORTLIST parameter for Manager is that there may not be enough numbers specified or freely available for use Frequently orphan processes occupy the ports desired by Manager You should either kill the zombie processes that occupy these ports or investigate ports that can be reserved for use with Oracle GoldenGate My Oracle Support Note 966097 1 http support oracle com has additional troubleshooting tips for network analysis with Oracle GoldenGate Network Data Transfer Issues Sometimes network latency issues cause a lag in data transfer by the Extract from the source system to the target system You can identify the latency issue by viewing the Extract checkpoint in the trail with the INFO EXTRACT SHOWCH GGSCI command as shown in this example INFO EXTRACT group SHOWCH GGSCI oracledba 13 info extract eataa showch EXTRACT EATAA Initialized 2011 03 16 00 50 Status STOPPED Checkpoint Lag 00 00 00 updated 00 17 15 ago Log Read Checkpoint Oracle Redo Logs 2011 03 16 00 50 47 Seqno 0 RBA O Current Checkpoint Detail Read Checkpoint 1 Oracle Redo Log Startup Checkpoint starting position in the data source Sequence 0 0 Timestamp 2011 03 16 00 50 47 000000 Redo
366. oldenGate 23 24 IFCONFIG command 231 IGNOREREPLICATES parameter 108 incompatible record errors Replicat process 238 incremental comparison 174 incremental data for large tables comparing 181 182 in flight status 182 in flight transactions 235 INFO lt extract name gt DETAIL command GGSCI 220 INFO ALL command 218 219 228 294 info all GGSCI prompt command 156 INFO command 64 68 69 82 268 271 294 295 info er command 284 INFO EXTRACT group command 228 229 INFO EXTRACT command 63 68 233 234 269 295 INFO EXTRACTS command 232 INFO EXTTRAIL command GGSCI 222 INFO EXTTRAIL trail file name command GGSCI 228 INFO MANAGER command 59 INFO MGR command 59 231 info rep command 284 INDEX INFO REPLICAT group command GGSCI 228 INFO REPLICAT command 82 233 234 236 248 271 INFO RMTTRAIL command GGSCI 222 236 initial compare 173 Initial Compares profile setting 180 initial load adding Replicat group 74 completion Microsoft SQL Server to Oracle replication 113 configuring Replicat parameter file 72 74 ASSUMETARGETDEFS parameter 73 MAP parameter 73 74 REPLICAT parameter 72 SETENV parameter 73 USERID parameter 73 Extract parameter file adding 71 configuring 71 prerequisites for 70 71 starting 74 verifying 74 75 Initial Load task 195 200 installation 5 31 281 283 checkpoint table 282 313 w
367. ommand It shows you if there were any collisions and how many for each type of DML statement See the section Verifying the Replicat for an example of the output Tip Techniques are available with specific load utilities that allow you to avoid using the HANDLECOLLISIONS parameter For example you can use the Oracle RMAN clone process for the initial load and start the GoldenGate Replicat to position after a specific Commit Sequence Number CSN in the trail corresponding to the Oracle System Change Number SCN from RMAN For more details download the white paper Oracle GoldenGate Best Practices Instantiation from an Oracle Source You can find the white paper at support oracle com Article ID 1276058 1 Next let s review at the ASSUMETARGETDEFS parameter ASSUMETARGETDEFS You can use the ASSUMETARGETDEFS parameter when your source and target tables are identical In the example the HR schema on the source and target is identical For GoldenGate to consider the tables identical they must have the same column names data types sizes and semantics and appear in the same order If the database tables being used on the source and target are different you need to generate a data definitions file instead Chapter 5 covers using a data definitions file MAP You use the MAP parameter to specify the mapping from your source tables to the target tables The source tables you specified previously in the Extract with the TABLE parameter
368. ompletion 113 preparing environment 113 sample heterogeneous replication 118 124 transferring source definition files to target 114 Microsoft SQL Server on Windows installation of Oracle GoldenGate 21 Microsoft Windows installation of Oracle GoldenGate 18 20 IBM DB2 UDB 23 24 requirements for Clustered environments 18 Sybase DBMS 23 Teradata RDBMS platform 21 22 migration zero downtime 263 278 INDEX MIN option 223 missing column errors 239 monitoring automating 162 169 checking disk space 168 169 checking memory and CPU scripts 168 checking process scripts 162 163 monitoring lag scripts 163 167 designing strategy for 153 155 importance of monitoring Extract process 155 maximum threshold 155 disk space 168 169 management and 284 287 automatically starting processes 287 discard file 286 generating interim statistics 286 GGSCI command shortcuts 284 285 OBEY files 285 286 purging old trail files regularly 287 reporting regularly on process health 286 memory and CPU scripts 168 monitoring lag scripts 163 167 process scripts 162 163 processes 155 162 all running 156 317 www it ebooks info INDEX checking logs 157 158 detail for Extract process 156 157 event and error logs 162 GoldenGate rate and redo log consumption rate 158 159 lags for each group 159 162 msconfig command 117 MSSQL2K8 environment 14 Multiple parallel Repli
369. on contained in this work The source code for this book is available to readers at www apress com You will need to answer questions pertaining to this book in order to successfully download the code www it ebooks info Contents About the Authors nennen nnn nnn nnne nnn About the Technical Reviewer xviii Acknowledgments xix Chapter 1 Introduction 1 Distributed Processing and Replication eeeeereererenreernnnn nnn 1 REPMCAUOM e 2 Oracle Advanced Replication iaces xsncnia 2 Oracle Streams HepliCaliOn ccc 3 Evolution and Oracle GoldenGate essent 4 Ur M n 4 Chapter 2 Installation nennen nensem nrc rcr 5 DOW AIOE CING the SOT ANG 5 Downloading from Oracle E DeliVEry csscsscsssscssssssscsssessesscsessssessssecsesecaeseesassesaesecaesassesassesanseeaneasaes 5 Downloading from OTN eese tentent tentent tnnt rnnt tente tentent toten tnn 9 Understanding Your Environment
370. on log whereas the Replicats are executing single threaded SQL commands to change data in the database on the target There may be situations however when an Extract needs some tuning For example in certain situations an Extract may need to fetch data from the database if there is a Large Object LOB column or if a column isn t in the transaction log In those cases you may want to isolate those tables into their own Extract group One common strategy for tuning GoldenGate is to create multiple parallel Extract or Replicat processing groups to balance the workload GoldenGate supports up to 300 processing groups per GoldenGate Manager As an example let s split the data pump Extract and Replicat for the HR schema Before you get started as a reminder Figure 7 1 shows the basic replication configuration with no parallel groups pp 0070777 ri Source Local D D Target Database Vui Data Pump x Replicat Database au sic eae een po E E Source Remote Trail Trail Figure 7 1 One way replication without parallel Extract or Replicat groups Now let s look at the same replication with two parallel data pump Extract and Replicat groups as shown in Figure 7 2 www it ebooks info CHAPTER 7 88 TUNING al Gal 1 edi 2 Figure 7 2 One way replication with parallel data pump Extract and Replicat groups Let s look at a specific example of how to implement parallel Extracts and Replicats on the HR schema
371. onfiguration issues 239 data pump errors 236 www it ebooks info data synchronization issues 238 239 Extract process 235 fetch failures 239 240 missing column errors 239 Replicat process 236 238 tables missing key constraints 239 source 34 35 235 target 39 40 tuning 152 289 290 data pump Extract group 247 254 adding 273 starting 276 278 stopping 275 277 DBLOGIN command 55 222 235 DBMS database management system configuration options 99 106 for ASM 102 103 DDL replication 103 106 for RAC 100 102 utilities loading with 76 DBMS vendor load utilities 270 DBOPTIONS parameter 57 DDL Data Definition Language replication 103 106 ddl setup sql 104 dedicated installation directories creating 282 INDEX dedicated users creating 281 DEFERREFCONST option 57 defgen command 98 DEFGEN utility 46 207 definition files generating 207 209 transferring source to target Microsoft SQL Server System 114 definition generators executing source on Oracle source system 114 DELETE REPLICAT command GGSCI 222 delivery 39 Desktop Client Director component 189 detail adding and monitoring 156 157 detail data command Logdump 300 Detail features Extract 206 DETAIL option 64 69 82 INFO EXTRACT group command 228 INFO EXTRACT command 233 INFO REPLICAT group command 228 INFO REPLICAT command 233 START EXTRACT extract name command 21
372. or Administrator Client e Oracle GoldenGate instances that will be monitored www it ebooks info CHAPTER 2 88 INSTALLATION The first step prior to performing the actual installation is to review the system requirements for the Oracle 11g GoldenGate Director software Let s take a look at the installation requirements for the Oracle GoldenGate Director Server System Requirements The following system configurations are necessary for the Oracle 11g GoldenGate Director installation to be performed successfully e Atleast 1GB of RAM more memory is always better e Minimum of 1 1 5 GB of disk space allocated for Director software e Dedicated port for Oracle GoldenGate Director Server The default port used is 7001 e Atleast 200 MB of disk space in the repository database for the Oracle GoldenGate Director Server to be used for schema objects Next you need to ensure that the particular platform used with Director is a supported configuration Following are the supported platforms Supported Platforms Oracle GoldenGate Director is available and supported on the following platforms e UNIX Solaris IBM AIX HP UX e Linux RedHat x86 x64 Oracle Enterprise Linux OEL e Windows x86 x64 For the Oracle GoldenGate Director Server you will also need to verify the following software has been installed and available for the Oracle GoldenGate Director Server before you perform the installation Software Requirements The following
373. or disk I O issues 159 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE e Processing data The Extract data pump or Replicat is busy grinding the data One of these processes may be the bottleneck It s normal if this status is temporary and goes away after a few seconds Using Write Checkpoint The next approach uses the Write Checkpoint n command to validate the lag GGSCI gt info extract extggs1 showch 10 Write Checkpoint 1 GGS Log Trail Current Checkpoint current write position Sequence 7 RBA 1411 Timestamp 2011 04 11 22 00 53 446000 Extract Trail C ggs dirdat Cc2 Write Checkpoint 1 GGS Log Trail Current Checkpoint current write position Sequence 6 RBA 4994 Timestamp 2011 04 10 22 51 25 477000 Extract Trail C ggs dirdat C2 Execute the above command on the data pump process and compare the Sequence RBA and timestamp If we expect the extract process to capture the data the sequence and RBA should increase in Extract Data Pump and Replicat processes If the sequence or RBA are stuck or increased slower than other process then that process may have lag issue Using the Trail File Generation Rate Another approach to monitor the lag is to note the current Sequence and file size for the Extract data pump and Replicat processes This chapter uses clnnnnnn for the data pump and c2nnnnnn for the Replicat the six ns in cl and c2 aren t the same each has its own sequ
374. orary database issue causes an abend a should fix it then the alert will go away However in rare cases if GoldenGate hangs this script won t catch the error You can use the lag alert script in the next section to catch almost all GoldenGate issues Monitoring Lag Scripts This lag monitoring script reports the overall lag from the source database to target database It s all on PL SQL and is easy to port to other databases You must run this script on the target database You can use a table with the most frequent transactions or you can use a dummy table source table and target table to track the lag If you use a dummy table you can update the same column with a timestamp every second using dbms job ora scheduled job Then calculate the time difference for the same table on the target side The difference is the lag value You can use two different schedules for this script e RunGGS GET LAG every 5 seconds to update the LAG HISTORY table e RunGGS GET LAG every hour to send information about the lag value to you so you know the alert is running If you don t get hourly information alert then you know something is wrong 163 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE This script is an independent monitoring process that doesn t tie to any GoldenGate process So it alerts us if any of the monitor points are broken in Figure 8 1 Some of the scenarios are as following e The GoldenGate administrator s
375. ord lag 7 seconds At EOF no more records to process GGSCI standbyserver gt STOP REPLICAT RHREMD2 Now you ve completed part 1 of the failover process The standby database has caught up with all the changes that are available from the primary database Now you can proceed to part 2 and start the standby Extract and switch the application to use the standby database as shown in Figure 12 10 257 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION 258 Start the Extract 1 Local Stendby Extract Primary Database Source Trail Database Application Start the Application 2 Figure 12 10 Disaster recovery unplanned failover part 2 Let s review the steps in part 2 from Figure 12 10 next 1 To begin capturing changes in preparation for the application switchover alter the Local Extract for the standby database to begin capturing changes now and then start it as follows GGSCI standbyserver gt ALTER EXTRACT LHREMD3 BEGIN NOW GGSCI standbyserver gt START EXTRACT LHREMD3 2 Prepare the standby database for the application and start the application connected to the standby database In preparation you may need to grant some SQL permissions to your application users Also remember to enable any triggers or cascade delete constraints you may have stopped while the standby was the replication target After completing part 2 you are done with the failover to the standby database and the us
376. ormance problem related to any specific database table or all tables Is the performance problem the result of increased workload or a system change Can the problem be consistently reproduced in a testing environment You may be surprised at the answers to these questions Perhaps the stakeholders consider the replication slowdown a low priority Maybe they re only seeing a slowdown at certain times of the day Or maybe the problem is the result of a system change or new business volume In any case it s important to get the stakeholders perspective and expectations You should document responses to the performance questions and obtain agreement from the stakeholders on the scope impact and expectations of the replication tuning effort Drill down on the GoldenGate processes that appear to be running slowly View the GoldenGate reports in the dirrpt directory and compare the processing rates to try to determine when the problem began Look at the rate and delta values and see when they started to decrease as shown in the following example from the Replicat report rpt rpt rpt rpt rpt RHREMD1 rpt 98650000 98660000 98670000 98680000 98690000 98700000 98710000 98720000 98730000 records records records records records records records records records processed processed processed processed processed processed processed processed processed as as as
377. ormulate your replication performance requirements It s best to keep the requirements simple and easy to understand Let s look at a few examples of performance requirements 128 www it ebooks info CHAPTER 7 88 TUNING The first example is for an important high volume Online Transaction Processing database involved in an active active replication topology It s critical to the business that the source and target databases be kept nearly synchronized all the time After working with the business users you ve agreed on the following performance requirement The replication lag must be less than 1 minute at least 80 percent of th e time If the replication lag exc eeds 1 minute a warning sho uld be issue d and there plication should be closely monitored If th e replication lag exceeds 5 minutes critical alert should be issued and the problem needs to be addressed immediately Let s look at another example of replication that isn t as critical In this example you re running replication from an OLTP database to a reporting database The reporting database is running on another highly utilized server with only limited server resources available to make the replication perform properly The reporting is used for offline analytical purposes and it isn t as critical to the business that the data is kept up to date In this case your performance requirement might look like this The replication lag from the OLTP database to the
378. ory and then abends 150 www it ebooks info CHAPTER 7 88 TUNING You can also run commands on the Replicat to confirm there is a network bottleneck The following example runs a status command on the Replicat as shown in the following example GGSCI targetserver gt send replicat RHREMD1 status Sending STATUS request to REPLICAT RHREMD1 Current status At EOF Sequence 17 RBA 2852 0 records in current transaction In the example the Replicat is in At EOF If the Replicat is At EOF and the Local Extract is processing data you know there is a bottleneck with the data pump and the network The Local Extract is extracting changes but there is nothing for the Replicat to process because of the network bottleneck Keep in mind that the Replicat will continue processing normally for a while if there is any leftover data in the target trail file until it hits the end of file Then it will stop processing because it isn t receiving data over the network in the target trail file from the data pump You can take several tuning approaches to help alleviate network performance issues The first approach is to create parallel data pumps as covered earlier in this chapter Parallel data pumps alleviate any network bandwidth limitations caused by using a single data pump process Second you can adjust some additional network tuning options on the RMTHOST parameter You see those next Tuning the RMTHOST Parameter The default GoldenGat
379. ote Trail Source Database Source Trail Figure 3 5 One way replication GoldenGate configuration Following is a description of the process as implemented in the figure 1 A Local Extract running on the source server extracts data and writes to the source trail 2 Adata pump is running on the source server This is optional but recommended The data pump reads data from the source trail and transmits it over the network to the remote trail 3 AReplicat running on the target updates the target database Multiple parallel Replicats can be configured if needed to improve performance A special case of one way replication is one way at a time replication as illustrated in Figure 3 6 In this scenario replication goes one way from the source to the target database and then is reversed from the target to the source The replication doesn t occur simultaneously like bidirectional replication which is discussed next One way at a time replication can be used for zero downtime database upgrades or migrations Zero downtime upgrades and migrations are covered in more detail in Chapter 13 Zero Downtime Migrations and Upgrades 41 www it ebooks info CHAPTER ARCHITECTURE Source Trail Switchover 4 Data Pump 6 Source Target D Local 1 Database Source Trail Figure 3 6 One way at a time replication GoldenGate configuration 1 i 1 Lye
380. ou see in your report file when BATCHSOL is activated Keep in mind that these tiny sample numbers based on the simple example in Figure 7 6 for illustration purposes BATCHSOL statistics Batch operations Batches Batches executed Queues Batches in error Normal mode operations Immediate flush operations PK collisions UK collisions FK collisions Thread batch groups Commits Rollbacks Queue flush calls NH BNOO OO OH mp N N U o9 Ops per batch Ops per batch executed Ops per queue Parallel batch rate 2 lt e gt 148 www it ebooks info CHAPTER 7 88 TUNING Let s review a couple of the fields in the BATCHSOL report Batch Operations are the total number of operations executed Batches are the number of batches created In this case you had three batches which you can see by referring to Figure 7 6 Batches in Error was 1 which tells you that you had an error while BATCHSOL was processing Because of the error Normal Mode Operations equals 1 because BATCHSOL rolls back and reverts to normal transaction processing mode operations when there is an error When BATCHSQL encounters processing errors you see the following type of messages in your GoldenGate logs 2011 03 22 17 12 14 WARNING OGG 00869 Aborting BATCHSOL transaction Database error 1 ORA 00001 unique constraint HR PK EMP ID violated 2011 03 22 17 12 15 WARNING 0GG 01137 BATCHSQL suspended continuing in norma
381. ou can use the GGSCI command INFO EXTRACT lt group gt DETAIL to locate the correct file If you want to store the parameter file in a different filesystem and directory then you need to use the PARAMS argument with the ADD EXTRACT command in GGSCI when you add the new Extract or if you ve already created the Extract group you can use the GGSCI command ALTER EXTRACT to change the parameter file location Another issue that poses problem with parameter configurations is related to file access permissions If the process group read and write permissions aren t granted correctly at either the operating system level or the database level then failures will occur when the Extract or Replicat process attempts to access the database on either the source or target system In addition if the file permissions aren t set correctly then errors will occur as well during Oracle GoldenGate operations For Windows you can check the permissions with the Windows Explorer graphical interface On Linux and UNIX platforms use the 1s 1command to check the status for read write and execute permissions You can use the chmod and chown UNIX and Linux commands to grant the required file system permissions to the Oracle GoldenGate parameter files as necessary A third configuration issue that occurs with Oracle GoldenGate process group parameter file configurations takes place when the required parameters are missing for either the Extract or Replicat This is often the
382. ou plan to implement Goldengate with your Oracle database environments Memory Requirements At least between 25 and 55 Mb of RAM memory is required for each GoldenGate Replicat and Extract process Oracle Goldengate supports up to 300 concurrent processes for Extract and Replicat per Goldengate instance As a rule of thumb you will need to take into consideration that at least 1 2 Extract processes and multiple Replicat processes will be required in addition to manager processes for a basic Oracle Goldengate installation The best way to assess total memory requirement is to run the GGSCI command to view the current report file and to examine the PROCESS AVAIL VM FROM OS min to determine if you have sufficient swap memory for your platform Next let s consider the disk requirements to install Oracle Goldengate 16 www it ebooks info CHAPTER 2 88 INSTALLATION Disk Space Requirements Following are some things you should do to ensure having enough disk space to support your Goldengate replication needs e Allocate at least 50 150 MB of disk space for the Oracle GoldenGate software binaries e Allocate 40 MB of disk space per instance of Goldengate for working directories and files per server For a basic configuration with Oracle Goldengate you will need to allocate 40 MB on the source and 40 MB on the target system for a total requirement of 80 MB of disk space e Allocate sufficient disk space to temporary files to accommodat
383. ou use the MAP parameter to specify the mapping from your source tables to the target tables The source tables were specified previously in the Extract with the TABLE parameter In the example you re using wildcarding to map all the tables in the HR schema on the source database to the same tables in the HR schema on the target database Similar to the TABLE parameter the MAP parameter is a complex parameter with many options that allow you to filter rows map columns transform data and so on Some of these options are covered in Chapter 5 For now let s keep it simple and just instruct GoldenGate to replicate all the changes for the HR schema You can do this easily by using the wildcard character an asterisk as in HR The wildcard tells GoldenGate to replicate all of the tables in the HR schema You could also list each of the tables 73 www it ebooks info CHAPTER 4 M BASIC REPLICATION 74 individually but using the wildcard is much easier You also assume that the source and target schemas match If they didn t you could use MAP parameter options such as COLMAP to map the different source and target columns Adding the Initial Load Replicat Before you can use the initial load Replicat you need to add the Replicat group You can do that using the ADD REPLICAT command as shown in the following example GGSCI targetserver gt ADD REPLICAT DHREMD1 SPECIALRUN The ADD REPLICAT command tells GoldenGate to add the Replicat
384. ould specify an additional parameter as shown in the following example to tell GoldenGate how to manage the secondary truncation points TRANLOGOPTIONS MANAGESECONDARYTRUNCATIONPOINT You can specify MANAGESECONDARYTRUNCATIONPOINT If you want GoldenGate to maintain a secondary truncation point You would specify this parameter if SQL Server replication wasn t running If SOL Server replication was running concurrently with GoldenGate then you would specify NOMANAGESECONDARYTRUNCATIONPOINT to allow SQL Server to manage the secondary truncation point TABLE Use the TABLE parameter to specify from which source database tables you wish GoldenGate to extract changes The TABLE parameter is a complex parameter it has many options that allow you to filter rows map columns transform data and so on Chapter 5 covers some of these advanced options For now let s keep it simple and just instruct GoldenGate to extract all of the table data changes for the HR schema You can do this easily by using a wildcard character the asterisk after the schema name In the example HR tells GoldenGate to extract all the tables in the HR schema You could also list each of the tables individually but using the wildcard is much quicker and easier The example also assumes that the source and target schemas match If they didn t you could use TABLE parameter options such as COLMAP to map the different source and target columns Tip Notice that the TABLE pa
385. our GoldenGate directories GGSCI server 7 show Parameter settings SET SUBDIRS ON SET DEBUG OFF Current directory gger ggs Using subdirectories for all process files Editor vi Reports rpt ggex ggs dirrpt 284 www it ebooks info CHAPTER 14 8 TIPS AND TRICKS Parameters prm gger ggs dirprm Stdout out gger ggs dirout Replicat Checkpoints cpr gger ggs dirchk Extract Checkpoints cpe gger ggs dirchk Process Status pcs gger ggs dirpcs SOL Scripts sql gger ggs dirsql Database Definitions def gger ggs dirdef Now let s display the history of the commands you ve executed by using the HISTORY command as shown in the following example GGSCI server 8 history GGSCI Command History info ext info rep info er info er info rep shell dir show history CON By using HISTORY combined with the command you can really save time You can use command to reexecute any command number contained in your history For example 7 reexecutes command number 7 which in this case is the SHOW command Finally you can use HELP to get help on any of the GoldenGate commands as shown here GGSCI server 9 gt help Next let s look at OBEY files which are another tool to make it even easier to manage GoldenGate Using OBEY Files You should store your frequently used commands in GoldenGate OBEY files OBEY files are command files similar to bat files in Wi
386. ownload oracle com docs cd E18101_01 index htm You can also find the installation guides for other DB2 platforms such as IBM DB2 z OS Installing Oracle GoldenGate Director 11g Oracle GoldenGate Director is an optional software module that provides an end to end management and administration console for Oracle GoldenGate environments Director uses a rich graphical interface similar to that found in network monitoring tools such as Tivoli and HP OpenView as well as Oracle Enterprise Manager OEM Director allows you to start and stop GoldenGate processes and to execute custom scripts as well The installation procedure is quite a bit different for this product in contrast to that used for the base Oracle GoldenGate transaction software Previous releases of the Oracle GoldenGate Director software had a different architecture and software installation process We will discuss only the current release of the Oracle GoldenGate Director software in this chapter Further details regarding how to configure Oracle GoldenGate Director will be discussed in upcoming chapters In addition we recommend that you consult the Oracle GoldenGate Director Administrator s Guide 11g Release 1 11 1 1 available online at http download oracle com docs cd E18101_01 doc 1111 e18482 pdf The installation process for Oracle 11g GoldenGate Director includes the following components e Oracle 11g Director Server Application e Monitor Agent e Director Client e Direct
387. perform better than the GoldenGate load On the other hand if you need to do complex transformations or are doing heterogeneous replication the GoldenGate initial load may work better The following sections discuss each load method Loading with GoldenGate You have several choices to make if you decide to perform the initial load with GoldenGate You can choose to have the Extract write the initial load records to a file that is then processed by the Replicat You can also have the Extract write out the initial load records in a format that can be processed by DBMS utilities like Oracle SQL Loader and the SQL Server Bulk Copy Program BCP Another method which is the one covered here is the Oracle GoldenGate direct load method With the Oracle direct load method no trail files are written to disk Instead GoldenGate extracts the data from the source database and sends it directly to a Replicat task on the target server which loads the data into the target database If you need further information you can find instructions for using the GoldenGate initial load methods in the Oracle GoldenGate Windows and UNIX Administrator s Guide In order to load the database using Oracle GoldenGate direct load you need to set up an initial load Extract and an initial load Replicat These are similar to the Extract and Replicat you already added for ongoing synchronization but with a few new parameters to let GoldenGate know these are to be used specifically
388. ple for Replicat RHREMP2 GGSCI standbyserver 2 STATS REP RHREMP2 Next let s add the GoldenGate processes for the replication from the standby database back to the primary database Configuring the Standby Extract Once Part 1 of your disaster recovery replication is complete and you are successfully replicating from the primary database to the standby database the next part is to add the standby Extract The standby Extract is similar to the primary Extract except it will be used to extract data to replicate from the standby www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION database back to the primary database if needed For now you re only adding the Extract Later you ll startit during switchover and failover activities Let s begin by configuring the parameter file for the standby Local Extract In our example you re configuring the Local Extract to capture all the SQL DML changes from the sample HR schema on the standby database Since you ve already used trails 13 and 14 let s use trails 15 and 16 for replication from the standby database back to the primary database Let s start by taking a look at the parameters for the Local Extract on the standby server GGSCI standbyserver 1 edit params LHREMP3 Extract LHREMP3 USERID GGER PASSWORD userpw ExtTrail dirdat 15 ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt LHREMP3 dsc Append DiscardRollover at 02 00 Table HR N
389. plicat on the standby database as shown in the following example GGSCI standbyserver START REPLICAT RHREMD2 Now you have completed Part 5 of the unplanned failover process and the databases are back to their original configuration The standby is now synchronized and ready for the next switchover or failover Summary In this chapter we covered how to set up and configure GoldenGate replication for disaster recovery purposes We showed how to configure GoldenGate to support a planned switchover scenario to minimize the application outage window during database maintenance We also showed how to configure GoldenGate to failover to the standby database and back to the primary during unplanned disasters If you want additional information on using GoldenGate to maintain a standby database you can also refer to the GoldenGate Windows and UNIX Administrator s Guide In the next chapter we ll cover another specific use of GoldenGate for zero downtime migrations and upgrades www it ebooks info 13 Zero Downtime Migration Replication Many IT organizations today are trapped for too long with old releases of software and old database hardware platforms because they can t tolerate the downtime involved with migrating to new software releases and hardware platforms Even planned database outages for busy online websites can result in lost revenue and make customers angry Oracle GoldenGate can help with these challenges by
390. pt EXTINIT dsc purge rmthost localhost mgrport 9000 rmttask replicat group REPINIT table gg director2 accountb Param Check Figure 10 10 It s good practice to validate the parameter file before running it 199 www it ebooks info CHAPTER 10 GOLDENGATE DIRECTOR Do the same map for Map for REPINIT Highlight the mapped table and then click Change Enter the Target table name in the Attributes box as shown in Figure 10 8 Now right click the EXTINIT Extract and choose Start as shown in Figure 10 11 Doing so spawns a GGSCI process to start the Extract Note that in Windows Director the spawned GGSCI process window sometimes doesn t exit properly if this happens go to the spawned GGSCI window and type Exit Or you can start EXTINIT in GGSCI The diagram refreshes the status in real time Oracle GoldenGate Director 11 1 1 0 0 loj xl File Edit Actions Controls Help leo agosa KDB OBA Bide Add New Data Sources z ij localhost gg Figure 10 11 Start the Extract from the Extract object Creating a One Way Replication When you create an Extract for the first time you need to do the following database tasks that Director Client can t do for you e Enable Supplemental Logging Minimum at the database level on the source database e Grant the proper privileges for Oracle GoldenGate users on the source and target If you want to log additional column change data you need to
391. pting the Trail Files nennen tnntn tantes 93 Adding Data Filtering and Mapping 94 xu 94 EC 95 j 96 Mapping Colin 97 Transforming 98 Oracle Specific DBMS Configuration Options esee 99 Configuring for Oracle RAG cuta tutu ta tata tu tune nt na ta nan natns i 100 Config ring for Oracle ASM irritat peri re tere rasa rena ve pa kae pepe esa cepe pesar eacus 102 Adding Oracle DDL Replication eene nennen tnter tnnt ntnns 103 Adding Bidirectional Replication rens 106 Excluding Transactions for Bidirectional Replication sse 108 Handling Conflict Resolution for Bidirectional Replication eene 109 Mur 110 www it ebooks info Chapter 6 Heterogeneous Replication 1er 111 Microsoft SQL Server to Oracle Replication eee 111 Preparing the Oracle
392. ption OK A OK AK A AK A OK kk k kkk KK Running with the following parameters Example Extract Parameter File SOURCEISTABLE 2011 03 14 00 00 33 INFO 0GG 01017 Wildcard resolution set to IMMEDIATE bec ause SOURCEISTABLE is used USERID ggs PASSWORD Source Context SourceModule ggdb ora sess SourceID gglib ggdbora ocisess c SourceFunction OCISESS try SourceLine 498 2011 03 14 00 00 33 ERROR 0066 00664 OCI Error during OCIServerAttach status 12560 ORA 12560 TNS protocol adapter error 2011 03 14 00 00 33 ERROR 0GG 01668 PROCESS ABENDING In this case Extract fails to run due to an Oracle protocol network error on the source database server The network error is caused by a failed Oracle 11g network listener TNS failing to start correctly You troubleshoot the error by examining the listener ora and tnsnames ora network configuration and correcting any errors Then restart both the Oracle listener and GoldenGate processes By correcting the source Oracle 11g listener protocol error you can successfully restart the Extract process Another useful way to investigate Extract failure problems is to run the GGSCI command INFO extract name DETAIL t
393. r example specific data types or structures may need to change if you re migrating from a SQL Server database to an Oracle database If you re simply migrating from an Oracle 10g to an Oracle 11g database you may have the exact same application data and data structures You may also be using Oracle GoldenGate to migrate other types of applications from old to new releases and the data structures in those cases may be different e Data in the old database must be kept current with the new database and there should be no replication lag at the time of cutover or fallback Before the cutover or fallback some lag can be tolerated but the lag must be eventually eliminated prior to the application cutover or fallback Any replication lag present at the time of cutover could cause a delay e Cutover to the new database and fallback from the new database to the old database must happen quickly to minimize any downtime Keep in mind that Oracle GoldenGate is only one piece the greater migration project and you need tested procedures in place for all parts of your migration project such as switching your application connections e After cutover to the new database the database roles are reversed and the new database becomes the replication source and the old database becomes the target This allows for migration fallback to the old database if needed e The time period permitted for the cutover and fallback procedures should be well understood For
394. racle GoldenGate environment This section shows you how to use these tracing tools to identify root cause of failures in Oracle GoldenGate The following tracing tools are discussed along with an example that showcases how to run TRACE e TLTRACE e TRACE TRACE2 Note f you require the use of tracing in Oracle GoldenGate you should first contact Oracle GoldenGate support while investigating failures within the Oracle GoldenGate environment so you can best deploy these tracing commands without adversely affecting your configuration Oracle GoldenGate Process Tracing with TLTRACE The TLTRACE parameter allows you to run a trace for database transaction log activity You can run a trace to show either basic or detailed information about records being processed To enable a TLTRACE session you need to add this parameter to the Extract or Replicat parameter file and restart the process 225 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE Using TRACE Parameters with Oracle GoldenGate The TRACE and TRACE2 parameters allow you to grab all details regarding Extract and Replicat processing The TRACE parameter provides detailed processing information for Oracle GoldenGate processes In contrast you use the TRACE2 parameter to identify database code segments where either the Extract or Replicat is taking the most time to process Troubleshooting Case Study with Oracle GoldenGate Let s walk through a case study tha
395. radata com Disk Requirements Teradata for Oracle GoldenGate has the following disk space requirements e 50 150 of space plus 40 MB of additional disk space for the Oracle GoldenGate binaries and directory structures for the Oracle GoldenGate software Additional guidelines for Teradata and Oracle GoldenGate can be found in the Oracle amp GoldenGate Teradata Installation and Setup Guide 11g Release 1 11 1 1 available online at http download oracle com docs cd E18101 01 index htm 22 www it ebooks info CHAPTER 2 88 INSTALLATION Installing Goldengate for Sybase on Windows and UNIX Like all database platforms Sybase presents unique requirements for installation of the Oracle GoldenGate software Disk space requirements for Sybase with Oracle GoldenGate are the same as those for Oracle with GoldenGate In addition to the disk space storage requirements you will need to grant operating system level privileges based on UNIX or Windows permissions so that the Sybase schema accounts and Oracle GoldenGate operating system level accounts can read and write at the database and operating system level between source and target systems involved with the Sybase and Oracle GoldenGate installation Otherwise errors will occur when the replication processes attempt to perform their functions The following database requirements must be completed as part of the Sybase and Oracle GoldenGate installation process e Configure DSQUERY variable
396. rail command only removes the GoldenGate checkpoints so you may also want to remove any leftover trail files from the operating system directory using O S commands to avoid any conflicts 3 Startthe GoldenGate Manager as shown in the following example GGSCI primaryserver START MANAGER Now that you have your GoldenGate files restored and the GoldenGate Local Extract ready on the primary database you can begin part 4 The next steps are to resynchronize the primary database from the standby database as shown in Figure 12 12 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION Performing the Initial Load 2 Start p Start the Data Pump ic E ue rg a Local Standby mm Replicat Eas Q Data Pump E 12 Primary Database Prepare for Load Remote Source Trail Trail Database Application 1 Stop the Application 3 Figure 12 12 Disaster recovery unplanned failover part 4 The part 4 steps for Figure 12 12 are described next 1 Disable any triggers or cascade delete constraints on the primary database 2 Perform the initial data load from the standby to the primary database This step can be done in hot mode while the standby database is running since the GoldenGate Extract will be capturing any DML changes made while the load is running 3 Once the initial load is done from the standby to the primary database stop the application running against the standby database At
397. rameter must end in a semicolon Next let s look at adding the Extract to the GoldenGate configuration 62 www it ebooks info CHAPTER 4 M BASIC REPLICATION Adding the Extract Now that you ve set up your Extract configuration parameters the next step is to add the Extract group You can do that using the following commands from GGSCI GGSCI sourceserver gt ADD EXTRACT LHREMD1 TRANLOG BEGIN NOW GGSCI sourceserver gt ADD EXTTRAIL dirdat li EXTRACT LHREMD1 MEGABYTES 100 The first command ADD EXTRACT adds the Extract using the configuration parameters defined in the last section After the Extract is added it establishes checkpoints in the source trail file and on the database transaction log to keep track of processing The TRANLOG parameter of the ADD EXTRACT command tells GoldenGate to use the database transaction log as its source In the Oracle example the redo logs are the source BEGIN NOW tells Extract to begin processing changes from the source database as soon as the Extract is started Optionally you can also instruct Extract to begin capturing changes at a specific timestamp or using a specific trail file number The ADD EXTTRAIL command adds the local Extract trail file assigns it to Extract LHREMDI and gives it a size of 100MB The default size of trail files is 10 You should size the trail files sufficiently large enough based on the transaction volume so that GoldenGate isn t creating new trail files
398. re going to use GoldenGate to keep the data synchronized ongoing after the load delete cascade constraints and triggers will need to remain disabled e You may want to drop indexes on your target tables during the load to speed up performance You can add these back when the load is complete This is optional Steps for Loading with DBMS Utilities You can use the following steps to load your target database using DBMS utilities in preparation for GoldenGate replication 1 Make sure you ve met the prerequisites described in the previous sections for starting replication and loading with the DBMS utilities 2 Startthe ongoing synchronization Local and data pump Extracts to capture changes while the load is running See the previous sections on starting the local and data pump Extracts for details 3 Execute the initial load using the DBMS utility you have chosen After the load is completely finished proceed with the next step 4 Startthe ongoing synchronization Replicat to apply changes made while the initial data load DBMS utility was running This also begins to apply new changes to keep the tables synchronized on an ongoing basis The next section goes over this Starting the Replicat After you ve loaded your data you can start the Replicat to apply the changes that were captured by the Extract while the load was running These changes were queued up in the trail files while the load was 76 www it ebooks info CHAPTER 4
399. red if needed to improve performance 43 www it ebooks info CHAPTER ARCHITECTURE 44 One drawback of bidirectional replication is that it can become complex Strategies must be developed for handling collisions and avoiding key conflicts For example one strategy might be that each database can process only a certain range of key values to avoid conflicts Broadcast Replication Broadcast replication as shown in Figure 3 9 is used to replicate data from a single source to multiple targets In this configuration the target databases are read only Figure 3 10 shows a typical GoldenGate configuration Broadcast replication is often used in environments where a copy of the production data is needed in several different geographic locations Distributing the data relieves the load on the primary database and gets the data closer to the end consumer of the data Target C D D Database Source gt Database Target Database Figure 3 9 Broadcast replication Local Source D Extract Database 1 Figure 3 10 Broadcast replication GoldenGate configuration Following are the key components from Figure 3 10 used to implement broadcast replication 1 ALocalExtract is running on the source servers 2 Two data pump Extracts are running on the source server pumping data to each of the target databases in parallel The data pumps can filter the data depending on the requirements of the target datab
400. repl For Teradata you don t need to identify the Replicat transaction however you must include the following SOLEXEC statements in the Replicat parameter file to execute a procedure that sets the Replicat session automatically at startup SQLEXEC SET SESSION OVERRIDE REPLICATION ON SOLEXEC COMMIT For SQL MX include the following in the local Extract parameter file to exclude the operations on the checkpoint table gger ckpt TRANLOGOPTIONS FILTERTABLE gger chkpt For Oracle 10g and later include the following local Extract parameter file to exclude the gger user TRANLOGOPTIONS EXCLUDEUSER gger www it ebooks info CHAPTER 5 ADVANCED FEATURES For Oracle 9i include the following TRACETABLE parameter in both the Extract and Replicat parameter files before any TABLE or MAP statements When an update occurs the Replicat updates the trace table This helps the Extract identify and exclude any GoldenGate transactions You have to create the trace table first using the ADD TRACETABLE command This example uses the trace table ggs_trace TRACETABLE ggs trace Next let s discuss how to handle conflict resolution for bidirectional replication Handling Conflict Resolution for Bidirectional Replication In a bidirectional configuration you must plan for identify and handle data conflicts Conflicts can occur if data belonging to a record with the same key value is updated on both databases at the same time Because GoldenGa
401. report ing database must be less than 1 hour at least 80 percent of the time If the replication lag exceeds 8 hours a warning sh ould be i ssued If the repl ication lag exceeds 24 ho urs critical aler t should be issued and the problem needs to be addressed within the following day You can see that each of these replication scenarios has very different performance requirements If you don t properly define the requirements up front you may spend all your time tuning the reporting replication when the active active replication is really the most critical Next let s look at creating a performance baseline Creating a Performance Baseline You should capture a baseline of the performance of your replication environment as soon as possible after your environment goes live A baselineis simply a set of performance metrics you can capture when your replication is running under normal conditions Normal conditions mean you don t want to collect your baseline metrics while any special or unusual processing is running or if there are any existing performance problems You don t want to capture the baseline when a performance spike is occurring After you ve captured the baseline you can use the baseline metrics for comparison purposes to identify possible performance problems You should also capture new baselines periodically particularly when there are major changes in your environment For example if a new server is added to your database
402. requisites are complete the first step in setting up zero downtime migration replication is to configure add and start the Local Extract Remember that you should start the Extract first to begin capturing changes made while the initial data load is running If you can afford to take an application outage and stop all change activity on the source database while the initial load is running then you can start the Extract after the initial load Refer to Chapter 4 for detailed steps for starting the Extract This section covers only the specific configuration for the zero downtime migration Extract 267 www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION 268 Let s begin by configuring the Local Extract In order to do this you first need to create a parameter file for the Extract Remember you re configuring the Local Extract to capture all the SQL DML changes from the example PAYROLL schema Let s look at the parameters for the Local Extract as shown in the following example GGSCI oldserver 1 edit params LPREMP1 Extract LPREMP1 USERID GGER PASSWORD userpw ExtTrail dirdat p1 ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt LPREMP1 dsc Append DiscardRollover at 02 00 Table PAYROLL TableExclude PAYROLL EXCLUDETABLE1 TableExclude PAYROLL EXCLUDETABLE2 Remember from the Oracle GoldenGate naming conventions for the Local Extract group name that the Lis used to designate t
403. rget HR EMPLOYEES Map HR JOBS Target HR JOBS Map HR JOB_HISTORY Target HR JOB_HISTORY Map HR LOCATIONS Target HR LOCATIONS Map HR DEPARTMENTS Target HR DEPARTMENTS 4 Add the parameter file for the new Replicat RHREMD2 GGSCI targetserver 1 gt edit params RHREMD2 Replicat RHREMD2 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD2 dsc Append DiscardRollover at 02 00 ON SUNDAY Map HR COUNTRIES Target HR COUNTRIES Map HR REGIONS Target HR REGIONS 5 Stop the existing data pump Extract PHREMDI and record EXTRBA and EXTSEQ from the info command You need those values when you add the new data pump Extract so it begins processing at the correct location in the Local Extract trail file From the info command shown in the following example you can see that EXTSEO is equal to the trail file number 18 EXTRBA is 8144 138 www it ebooks info CHAPTER 7 88 TUNING GGSCI sourceserver 1 gt stop ext PHREMD1 Sending STOP request to EXTRACT PHREMD1 Request processed GGSCI sourceserver 2 gt info ext PHREMD1 EXTRACT PHREMD1 Last Started 2011 03 20 13 09 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 10 ago Log Read Checkpoint File dirdat 11000018 2011 03 20 13 19 23 000000 RBA 8144 When the Replicat RHREMD1 has processed all the remaining changes
404. ring Strategy You need to master two factors in order to design a comprehensive monitoring strategy e Internal factor You need to understand how GoldenGate works internally You should have full control as a GoldenGate administrator If something wrong within the GoldenGate system you should be able to identify the issue and fix it yourself e External factor These are the systems that GoldenGate depends on In large organizations you may not have full access to these systems such as sophisticated network server and storage monitoring tools and utilities In some cases you can t even access the database monitoring tools if you aren t a DBA But you don t need to have full access to these systems to monitor your GoldenGate processes This chapter discusses the available monitoring techniques that are available to GoldenGate administrators In addition to these factors you also need to identify all the possible ways that GoldenGate can break The possible monitoring points for a simple GoldenGate configuration are shown in Figure 8 1 this chapter covers each of them If you can manage a simple configuration you can manage a complex configuration The number of moving parts in the GoldenGate system doesn t matter you just need to add more monitoring to each additional component For example with an active active configuration you need to replicate the monitoring activities on monitor points 1 to 7 in Figure 8 1 from the source 153
405. ronments Target A Conflict Ruleset Handler triggers DBMS_REPCAT COMPARE_OLD_VALUES DBMS_REPCAT SEND_OLD_VALUES Target B Figure 1 2 Conflict resolution for Oracle replication Oracle Streams Replication Oracle introduced the Streams replication solution in release 9 2 Streams resolved the deficiencies that plagued the older replication methods in previous releases It established a log based replication solution that mined the online redo logs for committed transactions on the source to move the data to downstream target systems In addition Streams introduced new background processes to maintain the communication and operations of replication activities within the Oracle ecosystem Streams further enhanced previous Oracle replication technologies by using advanced queuing technology to boost replication performance Oracle Advanced Queuing AQ is a technology designed with Streams and other Oracle technologies including Oracle Data Guard logical databases AQ provides a system of enqueues and dequeues based on the subscriber and publisher model for database communication It allows for robust messaging and processing of data transactions between source and target environments and lies at the heart of Oracle Streams processing www it ebooks info CHAPTER 1 88 INTRODUCTION Further details on how to set up and manage AQ are available online in the Oracle reference documentation at http tahiti oracle com
406. rs Here are the suggested steps in the GoldenGate performance and tuning methodology Steps 1 and 2 should be completed early in the project 1 Define the performance requirements 2 Create a performance baseline Steps 3 through 6 should be done as tuning issues are identified 3 Evaluate current performance 4 Determine the problem 127 www it ebooks info CHAPTER 7 88 TUNING 5 Design and implement a solution 6 Repeat steps 3 thru 5 as needed If you ve ever performed database performance and tuning these may look like familiar steps But each step has specific considerations in a GoldenGate replication environment This chapter explains and expands on the steps as you move along Now let s examine the first step in the methodology defining the performance requirements Defining the Performance Requirements The key to successfully meeting the performance expectations of your business users is to properly define the performance requirements Without performance requirements you never know if you ve successfully met the users expectations When you have the requirements defined and agreed on you can focus your tuning efforts on areas that don t conform to the requirements You should define the GoldenGate replication performance requirements early in the project before GoldenGate is even installed and configured You do this because these requirements can have a significant impact on how GoldenGate is set up and confi
407. running processes 156 checking logs 157 158 detail for Extract process 156 157 event and error logs 162 www it ebooks info GoldenGate rate and redo log consumption rate 158 159 lags for each group 159 162 reporting regularly on health of 286 tracing with TLTRACE parameter 225 Processing lag 130 Processing rate 130 profile startup file Oracle GoldenGate home directory 230 profiles 179 180 ps eflgrep smon command 228 PURGE option DISCARDFILE parameter 225 PURGEOLDEXTRACTS parameter 89 purgeoldextracts parameter 195 PURGEOLDEXTRACTS parameter 222 223 purging 222 223 Q QUOTA UNLIMITED privilege 27 R RAC Real Application Clusters configuration options for 100 102 connecting to database 100 101 defining threads 101 102 synchronizing nodes 100 INDEX RANGE filtering 142 RATE information 88 RBA relative byte address 224 234 Real Application Clusters configuration options for See RAC configuration options for Real Application Clusters Oracle RAC 21 records reporting discarded 89 redo logs error when Extract process unable to access 235 monitoring consumption rate 158 159 online current 157 references for further reading 291 292 relational database management system See Teradata RDBMS relational database management system MySQL RDBMS to Oracle replication 122 relative byte address 224 234 remote trails 34 38 39 R
408. s You should group together tables with referential integrity to one another into the same Extract process group You can also tune memory usage by the Extract in the event that the network latency issue has been addressed Oracle Database Issues with GoldenGate In an ideal world this book would cover the database configuration issues for each platform that supports Oracle GoldenGate Because the details are beyond the scope of this book I encourage you to consult the platform related Oracle GoldenGate documentation for third party RDBMS platforms online at http download oracle com docs cd E18101 01 index htm Oracle database related issues that cause errors with the Oracle GoldenGate operations include the following e Extract process can t read or access the online redo logs or Oracle database archive log files e Missing Oracle database archive logs e Out of sync Oracle database issues e Extract and Replicat failures with Oracle source and target database environments e Data pump errors www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE Extract Can t Access Oracle Database Archive and Redo Logs In the event that the Extract process can t find the requested archive or online redo log file on the source Oracle database it waits For instance you may encounter error messages from within Oracle GoldenGate such as this 2011 03 16 00 51 55 ERROR 0GG 00446 Error 5 Access is denied opening log file WINORA1
409. s 0 00 Total operations 6 00 Total insert collisions 2 00 You re now finished with setting up the basic replication Let s summarize what you ve learned Summary This chapter covered how to set up configure and verify basic GoldenGate replication You saw how to configure add and start a local Extract a data pump Extract and a Replicat You also examined the process for doing an initial data load using the Oracle GoldenGate direct load method and using the vendor DBMS load utilities The next chapter expands on these basic concepts and covers some advanced techniques to make your replication environment more secure and easier to manage and monitor 83 www it ebooks info 5 Advanced Features This chapter gives you an in depth understanding of some advanced features of GoldenGate replication These features are often necessary to make your replication ready to handle a real world production environment In addition you learn how to configure GoldenGate bidirectional replication to fully utilize both of your database servers in an active active replication configuration In Chapter 4 you learned how to set up and configure basic GoldenGate replication In this chapter you build on the basic replication and add features to make the replication more robust and easier to manage and monitor You also learn about some specific changes needed to make your replication work in specialized environments like Oracle Real Appl
410. s down you don t get any alerts that are running on that server With all your monitoring scripts in different places you can endure any system outages without missing any alerts Checking GoldenGate Process Scripts This monitoring script is low overhead you can run it every 5 minutes to check whether any GoldenGate process is up The script doesn t include a restart because most Manager parameters should include an auto restart Copy the block for each new process you want to monitor 162 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE bin ksh EMAIL_LIST shingchung aol com 5555555555 vtext com host hostname gghome ggs check ps ef grep extggsi grep v grep extggsi wc 1 check_num expr check if check_num 1 0 then echo GoldenGate Extract EXTGGS1 is down on host gt gghome extggs1 out tail 200 gghome dirrpt eextggsi rpt gt gt gghome extggs1 out mailx s GoldenGate EXTGGS1 Down on host EMAIL LIST lt gghome extggs1 out fi check ps ef grep dpggsi grep v grep dpggsi wc 1 check_num expr check if check_num 1 0 then echo GoldenGate Extract DPGGS1 is down on host gt gghome dpggs1 out tail 200 gghome dirrpt dpggsi rpt gghome dpggsi out mailx s GoldenGate Extract DPGGS1 Down on host EMAIL LIST lt gghome dpggsi out fi If there is any abend or even an inadvertent stop you get an alert e mail or page If a network issue or temp
411. s for identifying and handling conflicts but usually this depends on the application business rules Consult the Oracle GoldenGate Windows and UNIX Administrator s Guide for details on additional restrictions and considerations Now let s go though the steps for configuring bidirectional replication Excluding Transactions for Bidirectional Replication In a bidirectional configuration you have one or more Replicats running on each server applying SQL transactions You can imagine that an endless looping scenario of SQL transactions could occur if each Replicat always applied SQL from the other Replicat back and forth By default GoldenGate doesn t replicate SQL transactions applied by the Replicats in order to prevent this endless looping This is controlled by the GETREPLICATES or IGNOREREPLICATES parameter For most databases however you still need to identify to GoldenGate which Replicat user transactions to exclude You do so with different parameters depending on the DBMS as shown in the following examples For DB2 z OS and LUW Ingres and Sybase code use the following local Extract parameter file to exclude the gger user TRANLOGOPTIONS EXCLUDEUSER gger For Sybase include the following in the local Extract parameter file to exclude the default transaction name of ggs repl For SQL Server this parameter is required only if the Replicat transaction name is something other than the default ggs repl TRANLOGOPTIONS EXCLUDETRANS ggs
412. s to bring the standby database current and then the applications can use the standby database until the primary database is www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION repaired and brought back online Once the primary is back online you will need to go through some steps to resynchronize it with the standby In our example let s assume that you ve had a disaster and your primary database server is destroyed A new primary database server is on its way from the vendor and will be delivered and operational within 24 hours Until that time you will use the standby server as the primary database server Let s go through the steps using GoldenGate replication to failover from the primary database to the standby database After that you will go through the steps to fall back to the new primary database server after it gets installed and is operational Once again to keep it simple we ve divided the unplanned failover steps into five parts each with several steps Let s begin with part 1 as shown in Figure 12 4 Stop the Replicat 1 Standby Replicat Database Remote Trail Following is a description of the steps in part 1 as shown in Figure 12 9 Figure 12 9 Disaster recovery unplanned failover part 1 1 After verifying there is no lag stop the Replicat as shown in the following example GGSCI standbyserver gt LAG REPLICAT RHREMD2 Sending GETLAG request to REPLICAT RHREMD2 Last rec
413. saction logs e IBM DB2 UDB command line interface CLI must be available on the target database environment because the Oracle GoldenGate Replicat process uses this CLI to apply data to the target system Both IBM DB2 UDB Control Center graphical user interface GUI and command line interface CLI are highly recommended to be installed and available on source and target DB2 systems 23 www it ebooks info CHAPTER 2 INSTALLATION 24 After you have verified that the minimum disk space is available and that you have installed the required IBM DB2 UDB system tools you need to grant privileges to the Oracle GoldenGate user You will need to grant the SYSADM privilege or the DBADM database level access to the Oracle GoldenGate database schema account and operating system level account Note For IBM DB2 UDB you can grant permissions by executing the command GRANT DBADM ON DATABASE TO USER lt ggs_user gt or by using the IBM DB2 UDB Control Center utility Grant the following privileges to the Oracle GoldenGate schema database user within IBM DB2 UDB e Local CONNECT to the target IBM DB2 UDB database environment e SELECT on the IBM DB2 UDB system catalog views e SELECT INSERT UPDATE and DELETE on the target IBM DB2 UDB tables Additional information regarding the IBM DB2 UDB installation with GoldenGate is available in the Oracle GoldenGate DB2 LUW Installation and Setup Guide 11g Release 1 11 1 1 online at http d
414. saction logs for GoldenGate to be able to successfully replicate the change to the target database GoldenGate requires the key values in addition to the changed data in order for the GoldenGate Replicat to apply the changed data to the target database Adding supplemental logging to the source database tables ensures that the database logs the key values that GoldenGate needs in order to properly apply the updates on the target database You can use the GoldenGate ADD TRANDATA command to force the database to log primary key columns for all updates on your source database You don t need to enable supplemental logging on the target database for one way replication To add supplemental log data for key columns issue the following commands from the GoldenGate Command Interface GGSCT GGSCI sourceserver 1 dblogin userid gger password userpw Successfully logged into database If you re logging in to SQL Server the DBLOGIN command is slightly different as shown in the following example GGSCI sourceserver 24 dblogin sourcedb sqlserver userid sa password userpw Successfully logged into database SOURCEDB specifies the SQL Server ODBC data source and USERID specifies the database user ID If you re using Windows authentication you can leave off the USERID and PASSWORD options Next you can use the ADD TRANDATA command to enable the supplemental logging required by GoldenGate GGSCI sourceserver 2 add trandata hr employees Logg
415. se a SQL script such as the following Oracle database example to automatically generate your ALTER TABLE enable and disable commands for the cascade delete constraints and triggers Using a script to dynamically generate the commands from the database saves time and ensures more accuracy In this example the script prompts for the schema owner name set echo off set verify off set pagesize 2000 set linesize 250 set trim on set heading off set feedback off spool amp amp SCHEMA disable cascade delete constraints sql select alter table owner table disable constraint constraint_name from all constraints where delete rule CASCADE and owner amp SCHEMA 57 www it ebooks info CHAPTER 4 M BASIC REPLICATION spool off spool amp SCHEMA disable triggers sql select alter trigger owner trigger disable from all triggers where owner amp 5 spool off spool amp SCHEMA enable cascade delete constraints sql select alter table owner table name enable constraint constraint name from all constraints where delete rule CASCADE and owner amp SCHEMA spool off spool amp SCHEMA enable triggers sql select alter trigger owner trigger enable from all triggers where owner amp SCHEMA spool off The script generates four spool files which you can then execute in SQLPLUS to dis
416. se results Golden Gate is reading the redo log seqno 175 In RAC you see the sequence number for each node Seqno 9 is the remote trail file sequence number The send command gives you similar information about the source and target seqnos GGSCI 32 Send extract extggs1 status Sending STATUS request to EXTRACT EXTGGS1 EXTRACT EXTGGS1 PID 7220 Current status Recovery complete At EOF Current read position Sequence 175 RBA 36263936 Timestamp 2011 04 10 00 27 47 000000 Current write position Sequence 9 RBA 2964 Timestamp 2011 04 10 00 27 53 063000 Extract Trail C ggs dirdat C1 Checking the Current Oracle Online Redo Log As we know from previous section that extract EXTGGSI is reading the sequence 175 and we would like to know if the sequence 175 is the most current or not The following query tells us which redo log file sequence Oracle is processing SELECT A MEMBER B SEQUENCE B STATUS B FIRST_CHANGE B NEXT_CHANGE B ARCHIVED FROM V LOGFILE A V LOG B WHERE A GROUP amp B GROUP MEMBER SEQUENCE STATUS FIRST CHANGE NEXT CHANGER C ORCL REDOO3 LOG 174 INACTIVE 6591047 6621440 C ORCL REDOO2 LOG 173 INACTIVE 6560323 6591047 C ORCL REDOO1 LOG 175 URRENT 6621440 281474976710655 first_change and next_change are the Oracle SCN numbers The current redo log sequence is 175 EXTGGSI is also on sequence number 175 Both sequence are 175 and they are in sync So there is no lag
417. sic one way replication topology The primary database normally handles application activity except in the event of a switchover or failover We will refer to a switchover as a planned outage for things like software maintenance A failover is an unplanned outage where the primary database is totally lost for some period of time During a switchover or failover the standby database will become the primary database In order to meet the disaster recovery requirements we covered in the last section you ll have to ensure you can quickly and accurately switch the application activity from the primary database to the standby database and back Also you must capture any database changes made on the standby database to resynchronize with the primary database once it comes back online We ve indicated that with a dotted line in our illustration in Figure 12 1 Primary Standby Database Database Figure 12 1 Disaster recovery replication topology Now let s look at how to implement the disaster recovery replication requirements and topology using GoldenGate 243 www it ebooks info CHAPTER 12 8 DISASTER RECOVERY REPLICATION Setup Once the prerequisites are complete setting up disaster recovery replication from source to target can be accomplished in two parts and seven total steps as shown in Figures 12 2 and 12 3 Performing the Initial Load Starting the Starting the 9 Starting the PrimaryExtract Primary Data Pump Primar
418. soon as the data pump Extract is started You can also tell the data pump Extract to begin capturing changes at a specific timestamp or using a specific trail file number The ADD RMTTRAIL command adds the data pump Extract remote trail file assigns it to Extract PHREMDI and gives it a size of 100MB The default size of trail files is 10MB As with the Local Extract you should size the trail files sufficiently large enough based on the transaction volume so that GoldenGate isn t creating new trail files too often and slowing performance 67 www it ebooks info CHAPTER 4 M BASIC REPLICATION 68 Tip You can get help in GGSCI on any of the commands by entering the HELP command More detailed help is available on specific commands such as HELP ADD EXTRACT Starting and Stopping the Data Pump After adding the data pump Extract you need to start it to begin processing records from the source trail file as shown in this example GGSCI sourceserver gt START EXTRACT PHREMD1 If needed you can stop the data pump Extract using the STOP EXTRACT command For example you may need to make a change to the data pump Extract parameters To do that you stop the data pump Extract change the parameter file and then start the data pump Extract for the new changes to take effect Here is an example of the STOP EXTRACT command GGSCI sourceserver gt STOP EXTRACT PHREMD1 Let s look at a few commands to verify that the data pump Extract
419. ssthru mode for data pumps 288 PASSTHRU parameter 66 67 236 288 passwords encrypting 92 282 PATH variable 230 PAYROLL database 266 performance 287 290 creating baseline 129 131 defining requirements 128 129 evaluating current 131 132 fastest available storage 288 limiting number of extracts 288 parallel Replicat parameter files 288 Passthru mode for data pumps 288 running performance tests 287 tuning database 289 290 of Veridata tool 180 182 compare methods 180 comparing incremental data for large tables 181 182 disabling Confirm Out of Sync step 180 excluding columns 180 increasing number of threads 180 RTRIM on character fields 181 statistics 183 185 tuning profiles settings 180 persistently out of sync status 182 PING host name command 232 ping command 219 planned switchover disaster recovery replication for 251 256 platforms supported for Oracle GoldenGate Director 11g 25 PowerUser role name Tomcat Web Server Administration Tool screen 187 PR abbreviation 266 268 prerequisites for disaster recovery replication 241 242 Preview tab Compare Pair Mapping screen 178 Primary Extract 35 process failures 218 221 Extract process 219 221 without report diagnostics 221 process groups missing 228 process scripts monitoring 162 163 process paramfile path name prm syntax 221 processes automatically starting 287 monitoring 155 162 all
420. ssue to avoid data duplication is to set the KEYCOLS option for the TABLE and MAP statements in the Extract and Replicat parameter files along with the LIMITROWS option to prevent multiple rows from being updated Character Set Configuration Issues Because Oracle GoldenGate uses the database character set to perform data synchronization tasks you need to make sure the Oracle source and target character sets are the same to avoid synchronization errors If they re different then errors will result You can check the character sets by looking at the NLS_LANG environment variable on both the source and target Oracle database systems The source NLS_LANG value must be set to the same exact character set as the target system In addition the target Oracle database character set needs to be set as a superset of the source Oracle database system Missing Column Errors If columns are found to be missing between source and target Oracle GoldenGate won t replicate data correctly This has its root cause in two areas The first case arises if supplemental data hasn t been enabled on the source and target Oracle database environments correctly For Oracle you use the ADD TRANDATA command to add the required supplemental log data either at the database level or on a table by table basis Another cause of missing column errors occurs when the KEYCOLS parameter is used to perform column mapping transformations from the source to target environments If
421. stem where it lives then you may have errors with processing it One recommendation is to adjust the size of the discard file by using the following parameters www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE e DISCARDROLLOVER Specifies the parameters for aging of the discard file e MAXDISCARDRECS Limits the total number of errors written out to the discard file e PURGE option for DISCARDFILE Purges the discard file before writing out new information e MEGABYTES option for DISCARDFILE Sets the maximum size of the discard file The default size of the discard file is 1MB Can t Open the Discard File Oracle GoldenGate processes for the Extract and Replicat use the discard file during operational activities If there is an error in the location of the discard file the Extract or Replicat process will fail in GoldenGate as shown in the following error 2011 03 16 00 54 08 ERROR 0GG 01091 Unable to open file c win1itrgt dirrpt report dsc error 3 The system cannot find the path specified 2011 03 16 00 54 08 ERROR 0060 01668 PROCESS ABENDING Check the read and write permissions of the DISCARDFILE parameter you ve configured in the Extract and Replicat parameter file to ensure that the directory exists and has the correct permissions granted Using Trace Commands with Oracle GoldenGate Oracle GoldenGate provides several useful parameters that you can use to trace processes that fail within the O
422. t info extract eataa EXTRACT EATAA Initialized 2011 03 15 21 50 Status STOPPED Checkpoint Lag 00 00 00 updated 1435 45 44 ago Log Read Checkpoint Oracle Redo Logs 2011 03 15 21 50 47 0 RBA O You can also use the wildcard to show all Extracts GGSCI oracledba 14 gt info extract EXTRACT EATAA Initialized 2011 03 15 21 50 Status STOPPED Checkpoint Lag 00 00 00 updated 1435 45 09 ago Log Read Checkpoint Oracle Redo Logs 2011 03 15 21 50 47 Seqno 0 RBA To view status and details for local and remote trail files you can enter the INFO EXTTRAIL command GGSCI oracledba 16 gt info exttrail Extract Trail AA Extract EATAA Seqno 0 RBA 0 File Size 10M SEND The GGSCI command SEND is used to update the messaging facility to refresh the status reporting utilities In addition you can use it to send processing directives to an Oracle GoldenGate process group for example to instruct an Extract or a Replicat to perform a specific task Example GGSCI oracledba 37 gt send mgr getlag STATUS The STATUS command allows you to check the current status for process groups such as Replicat and Manager in the current Oracle GoldenGate environment Example GGSCI oracledba 40 gt status mgr Manager is running IP port oracledba 50001 You can find a complete list of commands in the Reference Guide available online as part of the Oracle GoldenGate documentation 295 www it ebooks info AP
423. t File dirdat 12000013 First Record RBA 2351 Add the new Replicat RHREMD3 and tell GoldenGate to begin processing at the EXTRBA and EXTSEO in the Local Extract trail 12 that you recorded in step 5 Once again if you don t tell GoldenGate to start at a specific location you can end up re replicating changes from the first trail and cause collisions in the Replicat GGSCI targetserver gt ADD REPLICAT RHREMD3 EXTTRAIL dirdat 12 EXTSEQNO 13 EXTRBA 2351 GGSCI targetserver info rep RHREMD3 REPLICAT X RHREMD3 Initialized 2011 03 20 22 51 Status STOPPED Checkpoint Lag 00 00 00 updated 00 00 01 ago Log Read Checkpoint File dirdat 12000013 First Record RBA 2351 Start the data pump Extract and begin sending new changes to the target server GGSCI sourceserver start ext PHREMD Sending START request to MANAGER EXTRACT PHREMD1 starting GGSCI sourceserver info ext PHREMD EXTRACT PHREMD1 Last Started 2011 03 20 22 51 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 06 ago Log Read Checkpoint File dirdat 11000018 First Record RBA 8144 Start all three of your Replicats to begin processing changes Run an info command to verify that they re running GGSCI targetserver start rep RHREMD Sending START request to MANAGER REPLICAT RHREMD1 starting Sending START request to MANAGER REPLICAT RHREMD2 starting 145 www it ebooks info CHAPTER 7 88 TUNING Sending START request to MANA
424. t Started 2011 01 15 13 53 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 08 ago Log Read Checkpoint Oracle Redo Logs 2011 01 17 22 55 08 Seqno 3261 RBA 7135232 If the Extract isn t running you can review the GoldenGate error log file and try to resolve the problem Often the error is something simple like incorrect spelling The error file is named ggserr log and is located in the GoldenGate software installation directory location In the example on Linux the error log is located in gger ggs ggserr log On Windows the GoldenGate error log based on the example is located in gger ggs ggserr log If your Extract won t start first check for obvious problems such as the wrong GoldenGate database user ID a bad password or typographical errors in your parameter file You should also verify that the source database is online and that you can access it using the GoldenGate database user and password Chapter 11 goes into more detail on troubleshooting You can see from the INFO command that the local Extract is RUNNING Checkpoint lagis the time delay between the last checkpoint written to the trail and the time when the record was processed by GoldenGate You currently don t have any checkpoint lag If you had a high checkpoint lag it could indicate a performance problem or perhaps the local Extract just catching up on a large volume of changes You can also see that Oracle redo logs are the source for the Extract the last read
425. t at the OTN site to receive access to free software for trial purposes white papers and webcasts from Oracle We also advise that you review the documentation for Oracle GoldenGate online at http download oracle com docs cd E18101 01 index htmto become familiar with the particular release notes for your database and operating system platform While the primary focus of this chapter will concentrate on Oracle database platform with GoldenGate we will provide coverage for additional RDBMS platforms such as MySQL and Teradata in terms of the installation requirements and guidelines for Oracle GoldenGate Downloading from Oracle E Delivery Let s get started and download the Oracle GoldenGate software from edelivery oracle com If you prefer to download from OTN then skip ahead to the next section where we show the use of http otn oracle com www it ebooks info CHAPTER 2 E INSTALLATION Figure 2 1 shows the results from a media search Choose the platform and version for your operating system to obtain the Oracle GoldenGate software as shown in the figure Choose Oracle Fusion Middleware as your platform Then specify your operating system Ele Edit View History Bookmarks Yahoo Tools Help jet form cal 2 Most Visited Getting Started Latest Headlines X O eR ime a DAEMON ze eig amp Search MS News Celebrity Video Weather Hotmail 48 Messenger Games Lifestyle EJ M
426. t command gives you a good indicator of GoldenGate s overall health As shown here Lag should be 0 and the Time Since Chkpt should be less than 10 seconds on normal operations If you have a long lag in your Extract the value in the Lag column is large or doesn t decrease GGSCI 28 info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING EXTRACT RUNNING DPGGS1 00 00 00 00 00 00 EXTRACT RUNNING EXTGGS1 00 00 00 00 00 04 REPLICAT RUNNING REPGGS1 00 00 00 00 00 09 Monitoring the Detail Extract To understand the current status of the Extract you can add detail to the info GGSIC command as follows GGSCI gt Info extract extggs1 detail EXTRACT EXTGGS1 Last Started 2011 04 09 22 36 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 02 ago Log Read Checkpoint Oracle Redo Logs 2011 04 10 00 16 47 Seqno 175 RBA 36263936 Target Extract Trails Remote Trail Name Seqno RBA Max MB C ggs dirdat C1 9 2964 5 Extract Source Begin End C MAPPNORADATANORCLNREDOO1 LOG 2011 04 09 22 02 2011 04 10 00 16 C APP ORADATA ORCL REDOO3 LOG 2011 04 09 20 14 2011 04 09 22 02 C MAPPNORADATANORCLNREDOO1 LOG Initialized 2011 04 09 20 14 Current directory C ggs Report file C ggs dirrpt EXTGGS1 rpt Parameter file C ggs dirprm EXTGGS1 prm Checkpoint file C ggs dirchk EXTGGS1 cpe Process file C ggs dirpcs EXTGGS1 pce Error log C ggs ggserr log www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE According to the
427. t ebooks info CHAPTER 5 ADVANCED FEATURES Working please wait Spooling to file ddl setup spool txt Using GGER as a GoldenGate schema name INITIALSETUP as a mode of installation Working please wait STATUS OF DDL REPLICATION SUCCESSFUL installation of DDL Replication software components Script complete Run the role setup script to set up the roles needed for DDL synchronization and grant the role to the gger database user SOL Qrole setup sql GGS Role setup script This script will drop and recreate the role GGS GGSUSER ROLE To use a different role name quit this script and then edit the params sql script to change the gg role parameter to the preferred name Do not run the script You will be prompted for the name of a schema for the GoldenGate database objects NOTE The schema must be created prior to running this script NOTE Stop all DDL replication before starting this installation Enter GoldenGate schema name gger Wrote file role setup set txt PL SOL procedure successfully completed Role setup script complete Grant this role to each user assigned to the Extract GGSCI and Manager processes by using the following SOL command GRANT GGS GGSUSER ROLE TO lt loggedUser gt where loggedUser is the user assigned to the GoldenGate processes SOL GRANT GGS GGSUSER ROLE TO gger Grant succeeded 105 www it ebooks info CHAPTER 5 ADVANCED FEATURES 106 6 Run the ddl enabl
428. t if you use column filtering in your data pump Extract you can no longer use passthru mode 95 www it ebooks info CHAPTER 5 ADVANCED FEATURES 96 Another example of column filtering is when you want all the columns in a table to be extracted except for a few columns Let s assume you want to extract the entire EMPLOYEES table except for the EMAIL column You can do that in the Extract with the COLSEXCEPT parameter as shown in the following example Table HR EMPLOYEES COLSEXCEPT email You may find that COLSEXCEPT is more convenient and efficient than using COLS especially if you only want to exclude a few columns from a large table Keep in mind that you can t exclude key columns with COLSEXCEPT In addition to filtering columns you can filter rows Filtering Rows The simplest way to filter rows is to add a WHERE clause to your TABLE statement on the Extract or to your MAP statement on the Replicat Let s look at an example of each First you add filtering to the Extract to only extract employees whose EMPLOYEE ID is less than 100 as shown in the following example Table HR EMPLOYEES WHERE EMPLOYEE_ID lt 100 Now let s add filtering to the Replicat Suppose you decide that you only want to replicate rows for your IT programmers You can add that filter to the Replicat as follows Map HR EMPLOYEES Target HR EMPLOYEES WHERE JOB ID IT PROG Tip Double quotes are required when filtering for da
429. t is writing to the trail file check the trail file size In Monitoring The Detail Extract section the trail file is c1000009 where 9 is the trail file seqno The trail file RBA is 2964 so you can use info extract extggs1 detail afew times to check if the RBA increases Or you can use ls 1s in Unix Linux or dir in Windows a few times to check the trail file size Monitoring the GoldenGate Rate and Redo Log Consumption Rate By default the stats command lists all the tables that have transactions You can limit the display using totalsonly by table and reportrate You can also reset the stats GGSCI 30 gt stats extggs1 Sending STATS request to EXTRACT EXTGGS1 Start of Statistics at 2011 04 10 00 05 46 Output to C ggs dirdat C1 Extracting from GGS SOURCE TABLE to GGS SOURCE TABLE Total statistics since 2011 04 09 22 36 18 Total inserts 0 00 Total updates 12 00 Total deletes 0 00 Total discards 0 00 Total operations 12 00 Daily statistics since 2011 04 10 00 00 00 No database operations have been performed Hourly statistics since 2011 04 10 00 00 00 No database operations have been performed 158 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE Latest statistics since 2011 04 09 22 36 18 Total inserts 0 00 Total updates 12 00 Total deletes 0 00 Total discards 0 00 Total operations 12 00 End of Statistics If you execute the following commands they shou
430. t shows how to use GGSCI commands and the error log to understand why a process failed in an Oracle database environment First you use the STATUS EXTRACT command which returns the following GGSCI ggs src 20 status extract extora EXTRACT EXTORA ABENDED Next use the VIEW GGSEVT command to drill deeper into the issue with the failed Extract process GGSCI ggs src 22 view ggsevt 2011 03 11 10 28 11 GGS INFO 399 GoldenGate Command Interpreter for Oracle GGSCI command admin start extract ggext 2011 03 11 10 38 15 GGS INFO 301 GoldenGate Manager for Oracle mgr prm Command received from GGSCI on host oracledba START EXTRACT GGEXT 2011 03 11 10 40 02 GGS INFO 310 GoldenGate Capture for Oracle ggext prm EXTRACT EXTORA starting 2011 03 11 10 41 11 GGS ERROR 501 GoldenGate Capture for Oracle extora prm Extract read error 13 Permission denied opening redo log C oracle arch 0001_0000000568 arc for sequence 258 2011 03 11 10 43 22 GGS ERROR 190 GoldenGate Capture for Oracle extora prm PROCESS ABENDING In this failure error message 501 indicates that the Extract user doesn t have the correct permissions to read the redo logs on the source Oracle database To fix this issue you grant read and write permissions on the source Oracle database so the Extract process can retrieve the redo log files After you ve granted these read and write permissions to the Extract user you need to stop the Manager process and exit GGSCI
431. t table for Oracle GoldenGate with a Replicat on Windows You run this from a Windows or UNIX Linux shell prompt window and not from within the GGSCI interface C Nggs trgt convchk repora ggs checkpoint EK EK EK SKK SKK KKK KKK Oracle GoldenGate Checkpoint Conversion Utility Version 11 1 1 0 0 Build 078 Windows optimized on Jul 28 2010 17 43 24 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved Starting at 2011 03 16 01 23 18 AK A A A OK o A A OK OK AK AK A AK A OK kkk OK OK OK KK Operating System Version Microsoft Windows Vista Business Edition on x86 Version 6 0 Build 6002 Service Pack 2 Process id 4880 Opening existing checkpoint file C ggs trgt dirchk REPORA cpr for group REPORA Checkpoint C ggs_trgt dirchk REPORA cpr backed up to C ggs_trgt dirchk REPORA cpr 1300252998 bak Updating checkpoint table in 8 0 2 compatible checkpoint file Successfully converted checkpoint to 8 0 2 compatible format Checkpoint conversion successful for group REPORA Once you ve fixed the checkpoint table for the Replicat you should be able to start the Replicat without errors as shown here GGSCI oracledba 14 info repora detail REPLICAT REPORA Last Started 2011 03 16 01 29 Status RUNNING
432. tRollover at 01 15 DiscardFile dirrpt LHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY ENCRYPTTRAIL ExtTrail dirdat 11 Table HR 93 www it ebooks info CHAPTER 5 ADVANCED FEATURES 94 Next is the enhanced Replicat parameter file GGSCI targetserver 1 gt edit params RHREMD1 Replicat RHREMD1 SETENV NLS LANG AMERICAN AMERICA AL32UTF8 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY DECRYPTTRAIL Map HR Target HR In the next section you add specialized filtering and mapping to your replication environment Adding Data Filtering and Mapping You can use GoldenGate to filter the tables columns and rows you re replicating You can also perform complex mapping of tables and columns from your source to target databases The following sections look at these topics and use the same sample HR table introduced in Chapter 4 Following is the layout of the table as a reminder Table EMPLOYEES Name Null Type EMPLOYEE_ID PK NOT NULL NUMBER 6 FIRST_NAME VARCHAR2 20 LAST NAME NOT NULL VARCHAR2 25 EMAIL NOT NULL VARCHAR2 25 PHONE NUMBER VARCHAR2 20 HIRE DATE NOT NULL DATE JOB ID NOT NULL VARCHAR2 10 SALARY NUMBER 8 2 COMMISSION PCT NUMBER 2 2 MANAGER ID NUMBER 6 DEPARTMENT ID NUMBER 4
433. ta values Another way you can perform complex filtering for numeric values is to use the GoldenGate FILTER parameter on either the Extract or the Replicat In addition you can use FILTER to restrict filtering to specific data manipulation language DML statements The following example filters only for employees whose monthly salary is greater than 1000 Map HR EMPLOYEES Target HR EMPLOYEES FILTER SALARY 12 1000 Let s look at another example using filtering based on the DML operation such as insert update or delete Suppose you only want to replicate deletes if the employee s monthly salary is less than 1000 You can set up a filter as shown next that only executes the filter for delete statements This example filters using the Extract TABLE parameter Table HR EMPLOYEES FILTER ON DELETE SALARY 12 lt 1000 You ve seen a few examples of the extensive capabilities of GoldenGate to filter tables rows and columns Now let s look at some advanced column mapping www it ebooks info CHAPTER 5 ADVANCED FEATURES Mapping Columns You may remember from Chapter 4 that the Replicat parameter file used the ASSUMETARGETDEFS parameter You used this parameter to tell GoldenGate that your source and target tables were identical But what if the source and target tables are different GoldenGate can easily handle these differences and even transform the data if needed Following is an enhanced version of the EMPLOYEES ta
434. tabase to the primary database include setting up and configuring the Local Extract data pump Extract and Replicat These Extracts and Replicats will be used later in switchover and failover scenarios The steps in Figure 12 3 are described as follows 1 Add the standby Extract Configure and add the standby Extract to be prepared to capture database changes in the event of a switchover or failover If there is an outage the standby Local Extract will already be in place and ready to start capturing changes 2 Add the standby data pump Extract Configure and add the standby data pump to be prepared to capture changes from the standby database in the event of an outage on the primary database 3 Add the standby Replicat Configure and add the standby Replicat to be prepared to begin applying changes back to the primary database if there is an outage Let s look at each of the disaster recovery replication steps beginnning with configuring the Local Extract in the next section Configuring the Local Extract for Disaster Recovery Let s begin by configuring the Local Extract In order to do this you will first need to create a parameter file for the Extract Remember we re configuring the Local Extract to capture all the SQL DML changes from the primary database HR schema for replication to the standby database HR schema We will keep an exact copy of the database data synchronized so it can be used for switchover or failover Let
435. tallation 27 Oracle GoldenGate installation 16 24 for IBM DB2 UDB on Windows and UNIX 23 on Linux and UNIX 20 for Microsoft SQL Server on Windows 21 on Microsoft Windows 18 20 and Oracle RAC considerations 21 requirements 16 18 for Sybase on Windows and UNIX 23 for Teradata on Windows and UNIX 21 22 Oracle GoldenGate Manager user account 222 Oracle GoldenGate root cause analysis 217 Oracle GoldenGate software 6 Oracle GoldenGate Troubleshooting Guide 300 Oracle GoldenGate user 228 Oracle GoldenGate Veridata installation 27 31 INDEX agent 28 29 system requirements 29 31 Oracle materialized views 174 Oracle protocol network error 220 Oracle RAC Real Application Clusters 21 Oracle Support certification matrix 112 Oracle Support Note 965703 1 237 Oracle Support Note 966097 1 232 Oracle Technology Network OTN site 9 12 291 292 ORACLE HOME 61 73 231 ORACLE SID 61 73 231 OS schedulers 178 OTN Oracle Technology Network site 9 12 291 292 ourceDB 61 Out Of Sync OOS report 173 owner field in Director Admin Tool 193 owner role in Director Admin Tool 193 P parallel files Replicat parameter 288 Parallel Replicat groups 142 Param File editor 189 parameter files 229 230 267 parameters finding in Director application 209 210 PARAMS argument 229 319 www it ebooks info INDEX 320 passthru mode 36 PASSTHRU mode 68 Pa
436. tallation directories are gger ggs ora10 for an Oracle 10 GoldenGate installation and c ggate sq12008 for a SQL Server 2008 installation Using a Checkpoint Table You should add a checkpoint table in your target database to keep track of checkpoints for your Replicats By default GoldenGate keeps track of checkpoints in a file on disk If you add a checkpoint file to the database GoldenGate makes the checkpoint part of the Replicat transaction which permits better recoverability in some situations The following example creates the checkpoint table GGSCI targetserver 1 dblogin userid gger password abc Successfully logged into database GGSCI targetserver 2 add checkpointtable gger chkpt Successfully created checkpoint table GGER CHKPT You should add the CHECKPOINTTABLE parameter to your GLOBALS file in the GoldenGate software installation directory for example gger ggs so the checkpoint table is used for all of your Replicats as shown here CHECKPOINTTABLE gger chkpt 282 www it ebooks info CHAPTER 14 8 TIPS AND TRICKS Verifying the Character Sets Sometimes you replicate between databases with different database character sets It s always a good idea to explicitly set the database character set inside the parameter file using the SETENV command This way you won t accidentally pick up a default character set from your environment settings that may be incorrect The following example shows how to set the dat
437. te is an asynchronous solution it s possible for these conflicts to occur The following are a few techniques you can use to avoid conflicts and handle conflict resolution Low latency You should make sure that the latency or lag between the two databases is minimized to prevent or at least minimize conflicts from occurring For example if a user on the first database updates Employee ID 100 s FIRST NAME to Bill and a few minutes later a different user on the second database updates the same employee s FIRST NAME to the correct name of William you want to make sure the correct name is applied to both databases If there is latency Bill may be applied after William If there is no latency the correct name of William will be applied in the proper order If there is significant latency then you may end up with more conflicts to resolve Key ranges You can assign specific key ranges to applications and ensure that they only update that key range on a specific database For example to avoid conflicts in the EMPLOYEES table you can assign EMPLOYEE IDs 0 1000000 to the first database and EMPLOYEE IDs 1000001 2000000 to the second database The HR application will be aware of the key ranges and code their DML accordingly to point to the correct database based on the key value Key identifiers Another technique is to add an additional key column to the existing key to identify the specific database where the record is owned or mastered For exampl
438. ted to show the Replicat configuration these reports can be updated periodically at user defined intervals with the latest processing statistics Target Database Like the source database the target database isn t really a GoldenGate component but rather a vendor database that is part of your infrastructure GoldenGate supports a variety of target databases Table 3 2 shows the databases GoldenGate supports as target databases as of version 11g Note Be sure to check the GoldenGate product documentation for the most current list of supported targets 39 www it ebooks info CHAPTER ARCHITECTURE Table 3 2 GoldenGate 11g Supported Target Databases Target Database c tree DB2 for iSeries DB2 for Linux UNIX Windows DB2 for z OS Generic ODBC MySQL Oracle SQL MX SQL Server Sybase TimesTen Manager The GoldenGate Manager process is used to manage all of the GoldenGate processes and resources A single Manager process runs on each server where GoldenGate is executing and processes commands from the GoldenGate Software Command Interface GGSCI The Manager process is the first GoldenGate process started The Manager then starts and stops each of the other GoldenGate processes manages the trail files and produces log files and reports Topologies and Use Cases Now that you re familiar with the GoldenGate architecture components let s discuss the various GoldenGate topologies and use cases Golden
439. tegrity In addition to replicating database DML data you can also replicate DDL changes and sequences using the Replicat if it s properly configured You can configure a special Replicat to apply data to initially load the target tables but this is typically done using DBMS utilities such as Data Pump for Oracle Just like the Extract you can configure Replicat as a single process or multiple parallel processes depending on the requirements Each Replicat process can act independently on different tables For example a single Replicat can apply all the changes for all the tables in a schema or you can create multiple Replicats and the divide the tables among them In some cases you may need to create multiple Replicat processes to improve performance You can stop and start each Replicat process independently Replicat can replicate data for an entire schema using wildcarding a single table or a subset of rows or columns for a table You can configure the Replicat to map the data from the source to the target database transform it and filter it to only replicate data meeting certain criteria For example Replicat can filter the Customer table to only replicate customer data where the name is equal to Jones Typically filtering is done by the Extract process and not the Replicat for performance reasons You can write any records that Replicat is unable to process to a discard file for problem resolution Reports can be automatically genera
440. tem In the event that the trail file wasn t created you can create a new trail file by executing the GGSCI command ADD EXTTRAIL trail file name EXTRACT group on the source system Replicat Experiences an Abend Failure on the Target System Recall from earlier chapters that Replicat processes live on the target database system and use a special database table called the checkpoint table 1f this table is accidentally deleted or corrupted Replicat processing fails One indication of this is the abend condition shown here www it ebooks info CHAPTER 11 8 TROUBLESHOOTING ORACLE GOLDENGATE 2011 03 16 01 13 58 ERROR 0GG 00446 Checkpoint table GGS CHECKPOINT does not exist Please create the table or recreate the REPORA group using the correct table 2011 03 16 01 13 58 ERROR OGG 01668 PROCESS ABENDING You may also see errors on the source system with the Extract such as GGS error 516 Extract read No data found selecting position from checkpoint table The solution to corrupted checkpoint errors is to drop and re create the checkpoint table on the target system for the Replicat or to use the convchk utility and then restart the Replicat My Oracle Support Note 965703 1 http support oracle com has tips on using the convchk utility The syntax for the convchk utility is convchk lt replicat group gt lt schema gt lt checkpoint table name gt Let s look at how to use the convchk utility to repair a corrupted checkpoin
441. th the latest database technology tips and information to share with the Oracle community Ben is also a polyglot being fluent in Spanish French Korean and English and he enjoys traveling to exotic places In his free time Ben enjoys racing in autocross events golf martial arts reading and cooking Steve Phillips is an Information Systems Architect with Hewlett Packard Enterprise Services and has been working with computer databases for over twenty five years Steve is an Oracle Certified Professional and a past presenter at Oracle Openworld and IOUG Steve holds a master s degree in Information Systems from the University of Texas at Dallas and currently lives in Plano Texas In his spare time Steve enjoys playing tennis coaching baseball and watching sports Richard Chung has 19 years of hands on experiences in all aspect of IT environments His recent focus is in the Business Intelligence area such as Reporting ETL and Data architecture Richard is Principal of BI and System Architect at Starwood Hotels and Resorts where he built the current data warehouse and reporting applications he also wrote highly sophisticated search engines and designed the ETL to populate the active data warehouse with near real time data Golden Gate is used to synchronize the critical business data around different cities in the US Before Starwood he worked at astartup dot com JP Morgan Chase and Ernst amp Young xvii www it ebooks info
442. the MINKEEPDAYS option The Manager process automatically deletes any trail files that have been fully processed that are older than two days Adding Automatic Process Startup and Restart Now you can add a couple of parameters to the Manager parameter file to try to avoid downtime for the Extracts and Replicats The AUTOSTART parameter as shown in the following example automatically starts all the Extracts and Replicats when the Manager starts The example uses wildcarding to tell the Manager to restart all Extracts and Replicats You can specify that you want to start only individual Extracts and Replicats if needed AutoStart ER In some cases an Extract or Replicat fails due to a temporary or intermittent problem such as a brief network outage The AUTORESTART parameter shown in the following example automatically tries to restart the Extracts and Replicats if they fail because of such a problem Note Use the AUTORESTART parameter with caution be sure you aren t unknowingly overlooking any problems you may prefer to have an Extract or Replicat process fail and then resolve the issue manually instead of using AUTORESTART AUTORESTART is best used as an exception for automatically resolving a known issue By default with AUTORESTART GoldenGate tries to restart the Extracts and Replicats two times after waiting two minutes between restarts Every two hours GoldenGate tries again You can adjust these defaults if needed Next
443. the Replicat for Zero Downtime Migration After you ve loaded your data you can start the Replicat to apply the changes that have been captured by the Extract while the initial data load was running These changes have been queued up in the trail files while the load was running and are waiting to be applied by the Replicat After you ve configured added and successfully started the Replicat the changes from the trail are applied to the target tables When the initial changes have been applied and there is no Oracle GoldenGate lag left the databases are fully synchronized At this point the Local and data pump Extracts and Replicat can continue to run and keep the old and new databases synchronized in real time with the ongoing changes The new database is ready and waiting for the application to cutover for the migration This section covers the various ongoing change Replicat configuration settings and options After that it discusses how to start the Replicat and verify that it s properly applying changes to the target Let s begin with configuring the Replicat Before you start to configure the Replicat you should go back and double check the Oracle GoldenGate prerequisites on the target server as covered in the earlier section Prerequisites Then you can begin configuring the Replicat In order to do this you first need to create a parameter file for the Replicat Remember you re configuring the Replicat to apply all the SQL DML
444. the SQL Exec parameter editor Without the Director Client interface it is very difficult to debug the SQL Exec statement and lead to many syntax errors Because there is no good IDE for editing parameter files you have to use vi or another generic text editor x Stored Procedure 66_ 1 OR2 ggs test MyFunction Query Logical Name oet userid Parameters param 1i userid If Filter afterfiter Exec Option map v All Params required v VARCHAR Len 200 Commit Db Changes Param Buffer Size 10000 Errors onore Trace Option all v ox ce Figure 10 27 SQL Exec editor for queries and PL SQL Director creates the following parameter code for you automatically after you click on the Ok button Without Director Client you have to type this word by word in a text editor in the exact same syntax extract EXTGGS1 userid gg director2 password gg director2 discardfile C ggs dirrpt EXTGGS1 dsc purge exttrail C ggs dirdat C1 table GG_DIRECTOR2 ACCOUNTB colmap usedefaults phone strext phone 2 6 fetchcolsexcept userid where userid hello keycols userid sqlexec spname GG DIRECTOR2 ggs test MyFunction id get userid params parami userid allparams required tokens DBUSERID GETENV ORAENVIRONMENT DBLOGINUSER SYSDATE GETENV ORATRANSACTION TIMESTAMP OPERATION QGETENV GGHEADER OPTYPE
445. the key team members and review the proposed scope and objectives for the replication You can even begin to sketch out the high level replication scenarios and flow so everyone understands what s involved After that you can formalize the rough draft business objectives share them with the team members and ask for feedback Finally the project scope and objectives should be reviewed and approved by the major stakeholders Here are just a few examples of business objectives you can achieve with GoldenGate e Minimize downtime while migrating from Oracle database 10g to Oracle 11g e Create an integrated database made up of sales customer and product data into new read only database for analysis and reporting 279 www it ebooks info CHAPTER 14 TIPS AND TRICKS e Offload intensive query and reporting activity from the primary OLTP database e Keep a copy of the primary database synchronized at an offsite location for disaster recovery purposes e Create an enterprise data warehouse for business users with near real time data integrated from multiple heterogeneous source databases e Minimize downtime while moving to a new platform and ensuring that a backout plan is available Understanding the Requirements You need to gain a solid understanding of the specific requirements that will drive the replication technical design Here are some of the factors that impact the replication design e Source and target configurations
446. the previous section 178 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA Creating a Profile Creating a profile is optional you can use the default profile for now We will go over this in detail later in the chapter Running the Veridata Job You can run the compare job by clicking the Run Execute job tab in the left panel on the Veridata Main screen Or you can use the command vericon exe job job_big table Run Execute Job Run Configuration i Description Job job big table b4 Job Profile normal bd Retrieve Compare Pair List Compare Pairs Compare Pairs 0 Filters Select Minen Row Previous Run Sena Previous Number Previous Number E od E T Status Partitions Duration Rows Compared Rows Out Of Sync Rei E Show 10 gt Items per page M 4 gt M Override Run Options Set Override Run Options Command Line To Be Used vericom exe job job big table Figure 9 5 Veridata Run Job The command line for this job is also shown in Figure 9 5 vericon exe job job big table for Windows Click the Run Job button to run the job The Run Job button text will change to Stop Job if it is running Click it if you want to stop the job You can also go to the Running Jobs or Finished Jobs panels to check the status If you get any error you can drill down the group and compare pairs to get the detail error messages 179 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDA
447. ther node in the cluster Starting with Oracle 11gR2 you can use an ASM Clustered File System ACFS for the GoldenGate software and files For more information refer to the white paper Oracle GoldenGate High Availability Using Oracle Clusterware available on the Oracle Technology Network at www oracle com technetwork index html Synchronizing the Nodes As part of the Oracle RAC installation you should make sure the date and time for each of the nodes in the cluster are synchronized Date and time synchronization is typically done using Network Time Protocol NTP Although synchronizing the date and time across nodes is part of the Oracle cluster installation it s especially important for replication because GoldenGate makes processing decisions based on the system date and time settings If the times are different on each node it can cause replication issues or abends You should also be aware of a few GoldenGate parameters that allow you to control the I O latency between the nodes You can use the THREADOPTIONS IOLATENCY to compensate for differences in I O latency between the nodes The default value is 1 5 seconds but you can increase this to 3 minutes if needed for significant latency Another option of the THREADOPTIONS parameter lets you adjust for delays caused by disk contention on the redo logs you can adjust the THREADOPTIONS MAXCOMMITPROPAGATIONDELAY up to 90 seconds to compensate for these delays The default is 3 seconds Co
448. there is a mismatch in source to target key column mapping errors occur You need to make sure you include either an existing key or a unique column in the KEYCOLS definition to resolve this error Filters and functions used in Extract and Replicat parameter files also cause these errors in the event that the correct existing key column isn t specified and found between the source and target mapping transformations given in the Extract and Replicat parameter files Fetch Failures Database fetch failures cause data synchronization errors if source row data is deleted during Oracle GoldenGate processing Another root cause of fetch failures occurs when undo retention for the Oracle source and target environments expires This causes the mandatory read consistent image of the data that the Extract process is looking for to disappear with the end result that the Oracle Snapshot Too Old 239 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE 240 message is displayed You can resolve this issue by adding the FETCHOPTIONS NOUSESNAPSHOT parameter to the Extract process group configuration This parameter makes the Extract fetch data from the table instead of going to the undo tablespace After you ve verified that Oracle GoldenGate has processed the record you need to remove FETCHOPTIONS so the Extract returns to fetching data from the undo tablespace You should also increase the amount of time that Oracle maintains data in the
449. time delay between the last checkpoint written to the trail and the time when the record was processed by GoldenGate You currently don t have any checkpoint lag If you had high checkpoint lag it could indicate a performance problem or the data pump Extract catching up on a large volume of changes Usually the data pump Extract in PASSTHRU mode doesn t experience lag because it s simply transferring data You can also see the source of your data pump as the 11 trail created by the local extract the time of the last read checkpoint and the RBA of the record in the trail file www it ebooks info CHAPTER 4 M BASIC REPLICATION You can add the DETAIL option to the INFO command to see even more information about the data pump Extract You may find the detail display helpful to see where the important files for the extract are located such as the parameter and report files This example shows the detail for the PHREMDI data pump Extract GGSCI sourceserver 3 gt info ext PHREMD1 detail EXTRACT PHREMD1 Last Started 2011 01 23 16 28 Status RUNNING Checkpoint Lag 00 00 00 updated 00 00 00 ago Log Read Checkpoint File dirdat 11000009 First Record RBA 3431 Target Extract Trails Remote Trail Name Seqno RBA Max MB dirdat 12 4 1151 100 Extract Source Begin End dirdat 11000009 2011 01 20 14 04 First Record dirdat 11000009 2011 01 14 10 22 2011 01 20 14 04 dirdat 11000007 2011 01 12 10 59 2011 01 14 10 22 dirdat 11000005 2011 01 07 08 38 20
450. too often and slowing performance Tip You should store your commands such as ADD EXTRACT an obey file in the diroby directory You can execute these from the GGSCI command prompt using the obey filename command This is good practice because the commands are readily available as a future reference or if you need to re execute the commands at some point Chapter 14 covers more about obey files Now let s look at how to start and stop the Extract Starting and Stopping the Extract After adding the Extract you need to start it to begin capturing changes as shown in the following example GGSCI sourceserver gt START EXTRACT LHREMD1 If needed you can stop the Extract using a similar method For example you may need to make a change to the GoldenGate parameters In this case you would stop the Extract make your changes to the parameters and then start the Extract for the new changes to take effect Here is an example of how to stop the LHREMDI Extract GGSCI sourceserver gt STOP EXTRACT LHREMD1 Verifying the Extract When the Extract has started you can verify that it s running using the INFO EXTRACT command You should see a status of RUNNING If you see a status of STOPPED or ABENDED there may be a problem In the following example you re checking the status of the LHREMDI Extract 63 www it ebooks info CHAPTER 4 M BASIC REPLICATION GGSCI sourceserver 2 gt info extract LHREMD1 EXTRACT LHREMD1 Las
451. tops the Extract inadvertently so GoldenGate is in the STOP state rather than ABEND e Someone deletes the Extract by mistake e The Extract process hangs The source server goes down e Someone accidentally turns off the monitoring shell scripts on the source target side If there are any issues with the CPU memory locks and so on that cause the database to slow down you get a high lag alert GG_LAG_HISOTRY table stores all the historical lag values You can use this table for reporting and integrating with other monitoring tools and ETLs Sample lag charts are shown in Figure 8 2 and Figure 8 3 CREATE TABLE GGS GG_LAG HISTORY AUDIT_TIME DATE LAG NUMBER Dummy Source table Map this table in the extract that you are monitoring CREATE TABLE GGS SOURCE_TABLE UPDATE_DATE DATE UPDATE_ID NUMBER Dummy Target table Map this in the replicat that you are replicating CREATE TABLE GGS TARGET TABLE UPDATE DATE DATE UPDATE ID NUMBER Schedule this to run every one or five seconds CREATE OR REPLACE PROCEDURE GGS GET LAG IS L MAX TIME DATE L LAG NUMBER BEGIN SELECT UPDATE DATE INTO L MAX TIME FROM TARGET TABLE 164 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE if target and source are in different timezone add timezone offset value in this calculation SELECT SYSDATE UPDATE DATE 24 60 60 INTO L_LAG FROM GGS TARGET TABLE IN
452. tract name Examine the output from the VIEW REPORT command in GGSCI the details provide clues to further investigation for resolving the Extract failure After you ve resolved the issue you need to restart the Extract process by executing the GGSCI START EXTRACT extract name command Issue the command with the DETAIL option as shown here GGSCI START EXTRACT extract name DETAIL 219 www it ebooks info CHAPTER 11 TROUBLESHOOTING ORACLE GOLDENGATE Now let s look with an example of an Extract process failing on an Oracle 11g source database server To view a report from GoldenGate execute the command GGSCI to enter the GoldenGate command line interface and then enter VIEW REPORT EATAA which references the name of the sample Extract process group GGSCI oracledba 3 view report eataa FRR AR A A OK A A OK A A KK K K K K K K KK KKK KKK KK SK KKK KKK KKK SK Oracle GoldenGate Capture for Oracle Version 11 1 1 0 0 Build 078 Windows optimized Oracle 11 on Jul 28 2010 18 00 34 Copyright C 1995 2010 Oracle and or its affiliates All rights reserved Starting at 2011 03 14 00 00 32 A A A A A A A OOK A E A A OK A A KK OK KK kkk KK Operating System Version Microsoft Windows Vista Business Edition on x86 Version 6 0 Build 6002 Service Pack 2 Process id 2524 Descri
453. ts on the new database before processing the Oracle GoldenGate changes from the old database GGSCI newserver START REPLICAT RPREMP1 The databases are back to their original configuration and ready for the next cutover attempt after any issues are resolved Summary This chapter covered how to set up and configure Oracle GoldenGate replication to support a zero downtime database migration scenario First you set up the basic Oracle GoldenGate processes for forward and reverse replication from the old to the new databases Then you saw how to configure Oracle GoldenGate to support the migration cutover to the new database and also fall back to the old database if needed For an additional reference on zero downtime migrations Oracle has a whitepaper titled Zero Downtime Database Upgrades Using Oracle GoldenGate available on the Oracle Technology Network website www oracle com technetwork middleware goldengate overview ggzerodowntimedatabaseupgrades 174928 pdf The next chapter covers some Oracle GoldenGate tips and tricks to help you set up and manage your Oracle GoldenGate replication environment in the best possible manner 278 www it ebooks info 14 Tips and Tricks Over the years of using GoldenGate we ve learned a number of helpful tips and tricks This chapter looks at some that you can use when you re setting up and managing GoldenGate replication These suggestions should save you time in getting yo
454. tting too large you roll it over every Sunday at 2 a m with the DISCARDROLLOVER parameter DiscardFile dirrpt LHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY You can choose to have GoldenGate report on discards for your local Extract data pump Extract and Replicat Purging Old Trail Files You can instruct GoldenGate to automatically purge your old trail files after they have been processed using the PURGEOLDEXTRACTS parameter Although you can add this parameter to individual Extracts and 89 www it ebooks info CHAPTER 5 ADVANCED FEATURES 90 Replicats it s best to add it to your Manager parameter file so purging can be managed centrally for all Extracts and Replicats Here is an example of the parameter you can add to your Manager parameter file to purge old trail files PurgeOldExtracts dirdat UseCheckpoints MinKeepDays 2 Let s take a closer look at the parameter and the options First you tell GoldenGate to purge the extracts in your dirdat directory where the trail files are stored The USECHECKPOINTS option tells GoldenGate to only delete the trail files after they have been fully processed by the Extracts and Replicats according to the checkpoint file You should almost always specify the USECHECKPOINTS option along with PURGEOLDEXTRACTS because otherwise your trail files may be deleted before they re processed Finally you keep the trail files for a minimum of two days after they have been processed per
455. two tables it s probably better to filter out the data for those at the Extract and send the Replicat only the data it needs You can see that you have to make some important design decisions as you configure your replication environment As a reminder let s review the basic Replicat MAP parameter from Chapter 4 In the following parameter example you replicate all the tables from the HR schema using the wildcard character Map HR Target HR If you want to replicate specific tables you can filter the tables using the Replicat much in the same way you filter them in the Extract The following example filters the tables to replicate only the EMPLOYEES and JOBS tables Map HR EMPLOYEES Target HR EMPLOYEES Map HR JOBS Target HR JOBS In addition to filtering tables you can also filter columns Let s look at that next Filtering Columns Suppose that instead of extracting all the columns in the EMPLOYEES table you re only interested in the EMPLOYEE_ID FIRST_NAME and LAST_NAME columns You can do this by adding the following to your Extract TABLE parameter Table HR EMPLOYEES COLS employee Id first name last name In the example all columns are ignored except the three columns listed in the COLS parameter Keep in mind that it s probably a good idea to include your key columns in the COLS list otherwise GoldenGate may choose all the columns listed as the key which may not be a good idea Note Keep in mind tha
456. u get an incompatible records with SCN mismatch error If you get this error either you lose the data or you need to use the Logdump utility to identify the problem records and patch the data manually In addition to the CSN seqno also plays an important role in monitoring and troubleshooting Seqno is the sequence number for the trail files each type of trail file has its own sequence number For example seqno 100 in the Extract trail file may not be the same as sequence number 100 in the Replicate or data pump even if they re in the same propagation process The naming convention for the trail file is 155 www it ebooks info CHAPTER 8 MONITORING ORACLE GOLDENGATE 156 aannnnnn where is two characters trail file prefix and the six ns are the seqno If GoldenGate uses up all the available seqno values it starts over from 000000 Technically you can have a maximum of one million trail files So set up the trail file size properly if you need to keep a lot of trail files The last key metric is the redo log sequence number Whenever you issue an alter extract begin now timestamp command GoldenGate searches the Oracle data dictionary to find out which redo log to begin reading from If no timestamp is given GoldenGate starts from the last checkpoint file Therefore it may take a while to start up an Extract especially with the NODYNAMICRESOLUTION parameter Monitoring Running Processes The info all GGSCI promp
457. uable when you re debugging or evaluating performance You should also include the REPORTCOUNT parameter with the RATE argument to display the number of records being processed and the processing rate In the following example Extract or Replicat parameters are configured to report the number of records processed every 15 minutes report the processing rate and generate interim runtime statistics every day at 12 00 ReportCount Every 15 Minutes Rate Report at 12 00 Now let s look at using a discard file for records GoldenGate can t process Using a Discard File You should specify a discard file in the Extract and Replicat parameter files GoldenGate uses this file to write any records that it can t process along with the error code You can use this file for debugging purposes in the event of failure You should check the discard file for changes periodically and resolve any discards as soon as possible Here is an example of using the discard file parameters in the Extract parameter We reviewed this in Chapter 5 Advanced and Bi Directional Features DiscardFile dirrpt LHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY Next let s review reporting on the health of your GoldenGate processes Reporting Regularly on Process Health You can use the GoldenGate Manager process to report on the health of the Extract and Replicat processes At a minimum you should consider including the following parameters in your Manager parameter f
458. unt points Trail files should be sized appropriately based on the amount of data captured Refer to Chapter 2 Installation for disk requirements and sizing formulas based on your transaction log activity www it ebooks info CHAPTER 14 8 TIPS AND TRICKS Tuning the Database Often GoldenGate performance issues aren t really caused by GoldenGate but instead by problems in the underlying DBMS Before starting replication you should make sure the performance of your database is adequate Then while your replication is running execute DBMS performance monitoring reports such as AWR for Oracle and verify that there are no outstanding performance issues If possible make sure your database tables have updated performance statistics and appropriate indexing on key columns used by GoldenGate Summary You can use the following checklist of questions to verify that you re taking advantage of the tips and tricks presented in this chapter e Requirements and planning Do you have a clear understanding of the business objectives and have they been approved by your stakeholders Do you have a solid understanding of the requirements Did you determine the appropriate replication topology e Installation and setup Did you create dedicated GoldenGate database and operating system users Did you encrypt the passwords in the GoldenGate parameter files Did you create a dedicated GoldenGate installation directory Are you using a dat
459. ups to ensure that it functions successfully The following versions of Microsoft SQL Server are supported with Oracle GoldenGate e SQL Server 2000 e SQL Server 2005 e SQL Server 2008 To verify that your specific Windows platform and MS SQL Server version is supported you can check the certification matrix online at My Oracle Support http support oracle com Software for non Oracle platforms can be obtained online from the E Delivery site at http edelivery oracle com Now that we have discussed the installation process for Oracle RDBMS with GoldenGate we will discuss how to prepare other database platforms with Oracle GoldenGate One thing to keep in mind is that the installation steps for the Oracle GoldenGate software are the same for other RDBMS platforms such as Teradata and Sybase However the subtle nuances lie in the preparation for these environments with Oracle GoldenGate Installing Goldengate for Teradata on Windows and UNIX Alas by default there is no built in replication facility within the Teradata RDBMS platform As such Teradata requires Goldengate to perform replication activities Teradata communicates with Oracle 21 www it ebooks info CHAPTER 2 INSTALLATION Goldengate via the Teradata Access Module TAM and through the Teradata Change Data Capture CDC on the source Teradata server Oracle Goldengate functions as the replication server and receives the transactions from the Teradata CDC source serv
460. ur environment up and running in the best possible fashion Keep in mind that these tips are based on many different experiences and they will generally give the best results However systems have different needs and requirements so these shortcuts may not be applicable in your case Also keep in mind that the tips and tricks will change over time as GoldenGate and the technology evolve You should always consult the standard Oracle GoldenGate product documentation which is the official guide for using the product Requirements and Planning This section is focused on preparing for your GoldenGate replication setup This process is often shortchanged or skipped altogether which later results in mistakes that end up costing more in the long run You should ensure that your project devotes proper time and focus to gathering and understanding requirements and that all the stakeholders approve of the requirements As with all projects carefully planning the replication project helps ensure its success Knowing the Business Objectives It s important for you to have a clear understanding of the business objectives for setting up GoldenGate The business objectives drive many of the technical decisions around the replication setup For example if the business objectives can be achieved with one way replication instead of bidirectional replication the technical complexity is greatly reduced You can start by having an informal whiteboard session with
461. us RUNNING Checkpoint Lag 00 00 00 updated 00 00 06 ago Log Read Checkpoint File c ggs dirdat c2000008 2011 04 12 18 28 41 860000 RBA 1455 In this case the Replicat is reading and processing seqno 8 on c2 The data pump is also writing seqno 8 on C2 so there is no lag However if that Replicat was reading a seqno less than 8 then there might be a database issue on the target database Recording the Lag Status in the ggserr log File You also can set the lag report frequency in the Manager parameter file LAGINFO MINUTES SECONDS HOURS example LAGINFOSECONDS 30 LAGREPORT MINUTES HOURS example LAGREPORTMINUTES 1 The Manager reports the lag information to the ggserr 1og file every 30 seconds and it checks the lag every minute Check ggserr for the lag reports Viewing Event and Error Logs The view reports group name command shows the most recent errors and events occurring to groups GGSCI gt view report extggs1 GGSCI view report mgr GGSCI gt view report repggs1 The detailed history is in ggserr log GoldenGate also logs errors in var log messages and to the Event Log in Windows Automating Monitoring Now that you know how to monitor GoldenGate processes manually it s time to automate them This section goes over a few monitoring scripts All the scripts are scalable to multiple Extracts and Replicats As a best practice the monitoring scripts should be in different systems because if the monitoring server i
462. utover Next let s add the fallback data pump Configuring the Fallback Data Pump for Zero Downtime Migration With the fallback Local Extract added you can configure and add the fallback data pump In this example you configure the data pump to read the p3 trail file written out by the fallback Local Extract named LPREMP2 and pump it over the TCP IP network to the old server to be processed by the Replicat www it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION From GGSCI you can edit and review the parameters for the data pump as shown here GGSCI newserver 1 edit params PPREMP2 Extract PPREMP2 PassThru ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt PPREMP2 dsc Append DiscardRollover at 02 00 RmtHost oldserver MgrPort 7809 RmtTrail dirdat p4 Table PAYROLL TableExclude PAYROLL EXCLUDETABLE1 TableExclude PAYROLL EXCLUDETABLE2 The next step is to add the data pump Extract group You can do that using the following commands GGSCI newserver gt ADD EXTRACT PPREMP2 EXTTRAILSOURCE dirdat p3 GGSCI newserver gt ADD RMTTRAIL dirdat p4 EXTRACT PPREMP2 MEGABYTES 500 You don t start the data pump until later so let s move on to adding the fallback Replicat Configuring the Fallback Replicat for Zero Downtime Migration Next you should add a fallback Replicat that will be ready to apply any changes to your old database after the application migration cutover
463. w how you arrive at that Extract name The specifies it s the Local Extract HREM specifies that you re using this Extract group for the human resources employee application in the HR schema The Dsignifies that you re running in the development environment And finally the 1 indicates this is the first Extract group for this schema and application You can add additional parallel Extract groups to improve performance as discussed further in Chapter 7 60 www it ebooks info CHAPTER 4 M BASIC REPLICATION SETENV You can use the SETENV parameter to override your default environment variable settings In the example you explicitly set NLS_LANG to AMERICAN AMERICA AL32UTF8 It s a good idea to set NLS_LANG and not let it unknowingly default to your environment setting If NLS_LANG is incorrect it could cause your replication to fail or replicate invalid data For the Local Extract NLS_LANG needs to match the source database character set You can run a SQL query such as the following for Oracle to determine your NLS character set SOL select from v nls_parameters where parameter like NLS CHARACTERSETZ PARAMETER VALUE NLS CHARACTERSET US7ASCII You can set other environment variables in addition to NLS LANG using the SETENV parameter For example with an Oracle database you can set ORACLE HOME and ORACLE SID as shown next In the example I chose instead to set the Oracle environment using the Oracle network TNS alias Sour
464. w it ebooks info INDEX 306 data mapping columns 97 99 generating data definitions file 97 98 transforming 98 99 data migration projects 175 Data Pump process 204 206 data pumps 36 38 65 70 283 adding 67 configuring 65 67 Extract parameter 66 PASSTHRU parameter 66 67 RMTHOST parameter 67 RMTTRAIL parameter 67 TABLE parameter 67 configuring for zero downtime migration 269 270 disaster recovery replication 246 247 errors with 236 fallback configuring for zero downtime migration 272 273 Passthru mode for 288 starting and stopping 68 verifying 68 70 Data requirements 50 280 data scripts 176 177 data source Director application 192 193 data validation 271 Data volume 50 128 280 database component Director Server and 190 database connections 175 176 Database error message 225 database management system See DBMS Sybase DBMS database migration projects 175 database performance statistics 130 database privileges and credentials granting to Oracle GoldenGate Director Server schema 27 Oracle GoldenGate Veridata 28 30 database server versions supported for Oracle GoldenGate 16 database sorting method 181 database level supplemental logging enabling 54 55 verifying 54 databases configuration Microsoft SQL Server to Oracle replication 113 114 source Oracle system 114 connecting to 100 101 issues with availability 227 228 character set c
465. ware location GoldenGate OS user ID password GoldenGate database user ID password Source server name Source manager port Source server OS release Source database name Source database vendor release Source schema Target server name Target manager port Target server OS release Target database vendor release Target database name Target schema Local extract name Data pump name Replicat name Local trail file Remote trail file gger ggs or gger ggs gguser userpw gger userpw sourceserver 7840 Linux 5 3 or Windows SourceDB Oracle 11 1 or SQL Server 2008 HR targetserver 7840 Linux 5 3 or Windows Oracle 11 1 or SQL Server 2008 TargetDB HR LHREMDI PHREMDI RHREMDI 11 12 The source and target schema used in the examples is the Human Resources or HR schema from the Oracle Database Sample Schemas Listed next is a description of the Employees table from the HR schema which is used in the examples 51 www it ebooks info CHAPTER 4 M BASIC REPLICATION 52 Table EMPLOYEES Name Null Type EMPLOYEE ID PK NOT NULL NUMBER 6 FIRST_NAME VARCHAR2 20 LAST NAME NOT NULL VARCHAR2 25 EMAIL NOT NULL VARCHAR2 25 PHONE NUMBER VARCHAR2 20 HIRE DATE NOT NULL DATE JOB ID NOT NULL VARCHAR2 10 SALARY NUMBER 8 2 COMMISSION PCT NUMBER 2 2 MANAGER ID NUMBER 6 DEPARTMENT ID NUMBER 4 One Way Replication Topology The topology covered in this chapter is one way replication
466. ways make a backup of your GoldenGate parameter files before you make any changes www it ebooks info Edit the existing data pump Extract parameter file PHREMDI to add the table filtering GGSCI sourceserver 1 edit params PHREMD1 Extract PHREMD1 PassThru RmtHost targetserver MgrPort 7809 RmtTrail dirdat 12 ReportCount Every 10000 Records Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt PHREMD1 dsc Append DiscardRollover at 01 00 Table HR EMPLOYEES Table HR JOBS Table HR JOB HISTORY Table HR LOCATIONS Table HR DEPARTMENTS Add the parameter file for the new data pump Extract PHREMD2 GGSCI sourceserver 1 gt edit params PHREMD2 Extract PHREMD2 PassThru RmtHost targetserver MgrPort 7809 RmtTrail dirdat 13 ReportCount Every 10000 Records Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt PHREMD2 dsc Append DiscardRollover at 01 00 www it ebooks info CHAPTER 7 88 TUNING 137 CHAPTER 7 TUNING Table HR COUNTRIES Table HR REGIONS 3 Edit the existing Replicat parameter file RHREMDI to add the table filtering GGSCI targetserver 1 edit params RHREMD1 Replicat RHREMD1 USERID GGER PASSWORD AACAAAAAAAAAAADAVHTDKHHCSCPIKAFB ENCRYPTKEY default AssumeTargetDefs ReportCount Every 30 Minutes Rate Report at 01 00 ReportRollover at 01 15 DiscardFile dirrpt RHREMD1 dsc Append DiscardRollover at 02 00 ON SUNDAY Map HR EMPLOYEES Ta
467. wntime Migration When you re successfully replicating from the old database to the new database the next part is to add the fallback Extract For now you re only adding the Extract Later you startit after the application cutover activities Let s begin by configuring the fallback Local Extract In the example you re configuring the Local Extract to capture all the SQL DML changes from the PAYROLL database on the new database except two tables that are being excluded Because you ve already used trail files p1 and p2 you can use trail files p3 and p4 for replication from the new database back to the old database Begin by reviewing the parameters for the Local Extract on the new server as shown in the following example GGSCI newserver 1 edit params LPREMP2 Extract LPREMP2 USERID GGER PASSWORD userpw ExtTrail dirdat p3 ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt LPREMP2 dsc Append DiscardRollover at 02 00 Table PAYROLL TableExclude PAYROLL EXCLUDETABLE1 TableExclude PAYROLL EXCLUDETABLE2 Next add the fallback Extract to the Oracle GoldenGate configuration so you re ready to start it after the application migration cutover You can do that using the following commands from GGSCI GGSCI newserver gt ADD EXTRACT LPREMP2 TRANLOG BEGIN NOW GGSCI newserver ADD EXTTRAIL dirdat p3 EXTRACT LPREMP2 MEGABYTES 500 You start the fallback Local Extract later after the application c
468. ww it ebooks info CHAPTER 13 ZERO DOWNTIME MIGRATION REPLICATION You should also run the STATS command on your Extract This shows if the Extract has processed any DML changes GGSCI oldserver 2 STATS EXT LPREMP1 You re now finished adding starting and verifying the Local Extract so let s move on to the next step starting the data pump Extract Configuring the Data Pump for Zero Downtime Migration When the Local Extract is started you can proceed with configuring adding and starting the data pump In this example you configure the data pump to read the p1 trail file written out by the Local Extract named LPREMPI on the old server and pump it over the TCP IP network to the target server to be processed by the Replicat on the new server From GGSCI you can edit the parameters for the data pump as shown in this example GGSCI oldserver 1 edit params PPREMP1 Extract PPREMP1 PassThru ReportCount Every 10000 Records Rate Report at 00 30 DiscardFile dirrpt PPREMP1 dsc Append DiscardRollover at 02 00 RmtHost newserver MgrPort 7809 RmtTrail dirdat p2 Table PAYROLL TableExclude PAYROLL EXCLUDETABLE1 TableExclude PAYROLL EXCLUDETABLE2 Now that you ve set up your data pump Extract configuration parameters the next step is to add the data pump Extract group on the old server You can do that using the commands shown in the following example GGSCI oldserver gt ADD EXTRACT PPREMP1 EXTTRAILSOURCE
469. ww it ebooks info INDEX 314 creating dedicated directory 282 creating dedicated users 281 data pump 283 developing naming standards 283 downloading software 5 12 from Oracle e delivery 5 8 from OTN 9 12 encrypting passwords 282 environment 12 14 instructions for 14 15 Oracle GoldenGate 16 24 for IBM DB2 UDB on Windows and UNIX 23 24 on Linux and UNIX operating system 20 for Microsoft SQL Server on Windows 21 on Microsoft Windows 18 20 and Oracle RAC considerations 21 requirements 16 18 for Sybase on Windows and UNIX 23 for Teradata on Windows and UNIX operating system 21 22 Oracle GoldenGate Director 11g 24 26 Oracle GoldenGate Director Server 27 Oracle GoldenGate Veridata 27 31 agent 28 29 system requirements 29 31 verifying character sets 283 in sync status 182 integration replication 44 45 interim statistics generating 286 IOSTAT command 150 IP address 231 IPCONFIG command Windows 231 J Java agent 173 jobs 178 179 K key constraints tables missing 239 key identifiers 109 key ranges parallel Extract and Replicat processes using 141 146 KEYCOLS option 238 239 KEYCOLS parameter 239 L L abbreviation 268 I2 trail file 136 13 trail file 136 140 lag command 130 lag scripts monitoring 163 167 www it ebooks info LAGCRITICAL parameter 286 LAGINFO parameter 286 lags monitoring for each group 159 162 getlag command
470. y Data distribution to multiple sites many Integration Ma ny to Active Passive Source only Data warehousing one 47 www it ebooks info 4 Basic Replication This chapter provides a good understanding of basic GoldenGate replication including an example of replicating a sample human resources schema You learn how to set up start and verify the basic replication configuration including the initial data load process You see examples in both Oracle Database on Linux and Microsoft SQL Server 2008 on Windows but keep in mind that most of the same GoldenGate commands and concepts apply to all of the supported platforms The chapter points out where a specific command or task applies only to an Oracle database or only to a SQL Server database Overview Now that you ve learned how to install the GoldenGate software and how the architectural components fit together you can begin setting up the replication This chapter covers setting up and configuring basic one way GoldenGate replication It discusses adding the local Extract data pump Extract and Replicat for ongoing change synchronization You also review initial data loading methods including using GoldenGate and native DBMS utilities More advanced configurations are covered in later chapters such as bidirectional replication in Chapter 5 heterogeneous replication in Chapter 6 and zero downtime migrations in Chapter 13 Keep in mind that the same concepts cover
471. y Replicat 1 Primary Local Database Extract Source Remote Database Trail Trail Application Figure 12 2 Disaster recovery replication setup part 1 Standby Database The steps are similar to setting up basic one way replication which we covered in Chapter 4 Part 1 includes the first four steps for setting up replication from the primary database to the standby database These four steps include initially loading the data and then keeping the data synchronized after the initial load Since we already covered these steps and concepts in Chapter 4 we won t cover them again here In the following sections in this chapter we ll provide the GoldenGate parameter files and only explain the new concepts for disaster recovery where applicable Adding the Adding the Adding the Standby Replicat Standby Pump Standby Extract 7 6 1 Primary a Qa Standby Database Replicat Extract Database Remote Source Database Trail Trail Application Figure 12 3 Disaster recovery reverse replication setup part 2 244 www it ebooks info CHAPTER 12 DISASTER RECOVERY REPLICATION Part 2 as shown in Figure 12 3 includes the last three steps for configuring replication back from the standby database to the primary should it be needed You will configure but not start the replication back from the standby database to the primary database The three steps for setting up replication from the standby da
472. y monthly ETL filtered data via partitioning or running the compare right after all the data is copied you can disable the Confirm Out Of Sync step in the profile settings Increasing the Number of Threads You can increase the number of threads on Veridata servers The default number of CPUs is four Since all comparisons take place on the server side Veridata compares are quite CPU intensive Verify the CPU usage during the compares before you increase the number of threads This can be set in both the Initial Compares and Confirmation Compares profile settings Compares Methods Using hash to compare is generally faster Hash is the default but if you have to use literally to get 10096 accuracy you can change the compare method in the Column Mapping Literal comparison is performed value for value as opposed to a comparison that uses a hash 180 www it ebooks info CHAPTER 9 ORACLE GOLDENGATE VERIDATA Right Trim on Character Fields By default Veridata has RTRIM on character fields on so your queries look like the following example You can turn RTRIM off in the Edit Database Connection setting Truncate Trailing Spaces When Comparing Values This reduces some CPU cycles to trim all the varchar columns in every single row SELECT SEQNO RTRIM OWNER RTRIM OBJECT NAME RTRIM SUBOBJECT NAME OBJECT ID DATA OBJECT ID RTRIM OBJECT TYPE TO CHAR CREATED yyyy MM DD hh24 mi ss TO CHAR LAST
473. y operating system environment variables for the database installed on the source and target systems For instance with Oracle make sure you check that the ORACLE SID and ORACLE HOME system variables are set to the correct instance name in the GoldenGate user profile You can use the env grep ORA command in Linux and UNIX to check that these Oracle environment variables are set to the correct values Network Configuration Issues with Oracle GoldenGate Oracle GoldenGate relies heavily on network operations to ensure data transfer and communications between source intermediary and target systems as part of its real time data replication functionality Latency issues and network outages adversely affect Oracle GoldenGate environments As such it s paramount that Oracle GoldenGate administrators develop a close partnership with the system administrator and network operations team to ensure maximum uptime and performance for the Oracle GoldenGate environment Network problems with Oracle GoldenGate fall into the following categories e Network access and connectivity e Network latency e Network availability and stability e Network data transfer issues Let s look into each of these key areas how to resolve issues One of the first errors I experienced years ago when I was a newbie to Oracle GoldenGate was a connection that refused errors that I noticed in the Extract report file Typically whenever you receive a common TCP IP error such as 4127 co

Expert Oracle GoldenGate

Contents

Download Pdf Manuals

Related Search

Related Contents