HP 3PAR corrupted rcopy delta - verify with checkrcopyvv
We suspected our 3PAR storage rcopy deltas were corrupt, and how should we verify? 3PAR will not tell us our We suspected our 3PAR storage rcopy deltas were corrupt, and how should we verify? 3PAR array won't really tell us its out of sync on such low levels, that our deltas have been corrupted.
There is an undocumented command named checkvvropy that can help us, and being undocumented there was no help text available.
Running this command gave me their useful message:
san001 cli% checkrcopyvv Insufficient number of arguments
Goggling returned one result again without any syntax or help, but I had some information from HP 3PAR support, and not its time to rectify this:
checkrcopyvv: Prerequisites: Host I/O is stopped to primary volume When rcopy fully Synced, stop the rcopy group. Else wait until synced. Run on primary rcopy site. If i/o cannot be stopped, you can run checkrcopyvv to compare the resync snapshot of the primary volume with the base volume on the secondary. These should be identical. ""checkrcopyvv -p pri_snap target:sec_base_vv"". This approach cannot be taken during a periodic resync.
So, I have an rcopy group:
san001 cli% showrcopy groups TITPRD
Remote Copy System Information
Status: Started, Normal
Group Information
Name Target Status Role Mode Options
TITPRD its-san002 Started Primary Periodic Last-Sync 2013-10-31 02:27:33 CET , Period 1d, auto_recover, over_per_alert
LocalVV ID RemoteVV ID SyncStatus LastSyncTime
TITPRD_PRI_001 439 TITPRD_SEC_001 7211 Synced 2013-10-31 02:29:14 CET
Let's kick this off and see how we go. I have changed our database names, of course.
san001 cli% showvv -p -prov snp -copyof TIT*
san001 cli% showrcopy groups TITPRD
san001 cli% stoprcopygroup TITPRD
san001 cli% checkrcopyvv -p pri_snap target:sec_base_vv
checkrcopyvv -p rcpy.13.439.1 san002:TITPRD_SEC_001
# start Oct 31 09:28
# end Thu Oct 31 10:22:24 CET 2013
# VSize 131072 MB = 131 GB
Consistency check passed
Difference map - each chunk is 32977kb. ""X"" indicates a differing segment:
0- 1648850kb: ..................................................
1648850- 3297700kb: ..................................................
3297700- 4946550kb: ..................................................
4946550- 6595400kb: ..................................................
6595400- 8244250kb: ..................................................
8244250- 9893100kb: ..................................................
9893100- 11541950kb: ..................................................
11541950- 13190800kb: ..................................................
13190800- 14839650kb: ..................................................
14839650- 16488500kb: ..................................................
16488500- 18137350kb: ..................................................
18137350- 19786200kb: ..................................................
19786200- 21435050kb: ..................................................
21435050- 23083900kb: ..................................................
23083900- 24732750kb: ..................................................
24732750- 26381600kb: ..................................................
26381600- 28030450kb: ..................................................
28030450- 29679300kb: ..................................................
29679300- 31328150kb: ..................................................
31328150- 32977000kb: ..................................................
32977000- 34625850kb: ..................................................
34625850- 36274700kb: ..................................................
36274700- 37923550kb: ..................................................
37923550- 39572400kb: ..................................................
39572400- 41221250kb: ..................................................
41221250- 42870100kb: ..................................................
42870100- 44518950kb: ..................................................
44518950- 46167800kb: ..................................................
46167800- 47816650kb: ..................................................
47816650- 49465500kb: ..................................................
49465500- 51114350kb: ..................................................
51114350- 52763200kb: ..................................................
52763200- 54412050kb: ..................................................
54412050- 56060900kb: ..................................................
56060900- 57709750kb: ..................................................
57709750- 59358600kb: ..................................................
59358600- 61007450kb: ..................................................
61007450- 62656300kb: ..................................................
62656300- 64305150kb: ..................................................
64305150- 65954000kb: ..................................................
65954000- 67602850kb: ..................................................
67602850- 69251700kb: ..................................................
69251700- 70900550kb: ..................................................
70900550- 72549400kb: ..................................................
72549400- 74198250kb: ..................................................
74198250- 75847100kb: ..................................................
75847100- 77495950kb: ..................................................
77495950- 79144800kb: ..................................................
79144800- 80793650kb: ..................................................
80793650- 82442500kb: ..................................................
82442500- 84091350kb: ..................................................
84091350- 85740200kb: ..................................................
85740200- 87389050kb: ..................................................
87389050- 89037900kb: ..................................................
89037900- 90686750kb: ..................................................
90686750- 92335600kb: ..................................................
95633300- 97282150kb: ..................................................
97282150- 98931000kb: ..................................................
98931000- 100579850kb: ..................................................
100579850- 102228700kb: ..................................................
102228700- 103877550kb: ..................................................
103877550- 105526400kb: ..................................................
105526400- 107175250kb: ..................................................
107175250- 108824100kb: ..................................................
108824100- 110472950kb: ..................................................
110472950- 112121800kb: ..................................................
112121800- 113770650kb: ..................................................
113770650- 115419500kb: ..................................................
115419500- 117068350kb: ..................................................
117068350- 118717200kb: ..................................................
118717200- 120366050kb: ..................................................
120366050- 122014900kb: ..................................................
122014900- 123663750kb: ..................................................
123663750- 125312600kb: ..................................................
125312600- 126961450kb: ..................................................
126961450- 128610300kb: ..................................................
128610300- 130259150kb: ..................................................
130259150- 131908000kb: ..................................................
131908000- 133556850kb: ..................................................
133556850- 135205700kb: ....................
san001 cli%
There were no Xs listed so our deltas are fine. This took an hour for our 130GiB Oracle database! checkrcopyvv should be used for troubleshooting purposes only for HP technical support, and not for us users, which is why I've blogged this :) Time for us to dig deeper and luckily the Storage Processor runs RedHat and the nodes run Debian.. .. Hmm.