HP 3PAR corrupted rcopy delta - verify with checkrcopyvv

We suspected our 3PAR storage rcopy deltas were corrupt, and how should we verify? 3PAR will not tell us our We suspected our 3PAR storage rcopy deltas were corrupt, and how should we verify? 3PAR array won't really tell us its out of sync on such low levels, that our deltas have been corrupted.

There is an undocumented command named checkvvropy that can help us, and being undocumented there was no help text available.

Running this command gave me their useful message:

    san001 cli% checkrcopyvv
    Insufficient number of arguments

Goggling returned one result again without any syntax or help, but I had some information from HP 3PAR support, and not its time to rectify this:

    checkrcopyvv:
    Prerequisites: Host I/O is stopped to primary volume
    When rcopy fully Synced, stop the rcopy group. Else wait until synced.
    Run on primary rcopy site.
    If i/o cannot be stopped, you can run checkrcopyvv to compare the resync snapshot of the primary volume with the base volume on the secondary. These should be identical. ""checkrcopyvv -p pri_snap target:sec_base_vv"". This approach cannot be  taken during a periodic resync.

So, I have an rcopy group:

    san001 cli% showrcopy groups TITPRD
    Remote Copy System Information
    Status: Started, Normal
    Group Information
    Name         Target     Status   Role       Mode     Options
    TITPRD       its-san002 Started  Primary    Periodic Last-Sync 2013-10-31 02:27:33 CET , Period 1d, auto_recover, over_per_alert
      LocalVV        ID   RemoteVV       ID   SyncStatus    LastSyncTime
      TITPRD_PRI_001  439 TITPRD_SEC_001 7211 Synced        2013-10-31 02:29:14 CET

Let's kick this off and see how we go. I have changed our database names, of course.

    san001 cli% showvv -p -prov snp -copyof TIT*
    san001 cli% showrcopy groups TITPRD
    san001 cli% stoprcopygroup TITPRD
    san001 cli% checkrcopyvv -p pri_snap target:sec_base_vv
    checkrcopyvv -p rcpy.13.439.1 san002:TITPRD_SEC_001
    # start  Oct 31 09:28
    # end   Thu Oct 31 10:22:24 CET 2013
    # VSize 131072 MB = 131 GB
    Consistency check passed
    Difference map - each chunk is 32977kb. ""X"" indicates a differing segment:
               0-     1648850kb: ..................................................
         1648850-     3297700kb: ..................................................
         3297700-     4946550kb: ..................................................
         4946550-     6595400kb: ..................................................
         6595400-     8244250kb: ..................................................
         8244250-     9893100kb: ..................................................
         9893100-    11541950kb: ..................................................
        11541950-    13190800kb: ..................................................
        13190800-    14839650kb: ..................................................
        14839650-    16488500kb: ..................................................
        16488500-    18137350kb: ..................................................
        18137350-    19786200kb: ..................................................
        19786200-    21435050kb: ..................................................
        21435050-    23083900kb: ..................................................
        23083900-    24732750kb: ..................................................
        24732750-    26381600kb: ..................................................
        26381600-    28030450kb: ..................................................
        28030450-    29679300kb: ..................................................
        29679300-    31328150kb: ..................................................
        31328150-    32977000kb: ..................................................
        32977000-    34625850kb: ..................................................
        34625850-    36274700kb: ..................................................
        36274700-    37923550kb: ..................................................
        37923550-    39572400kb: ..................................................
        39572400-    41221250kb: ..................................................
        41221250-    42870100kb: ..................................................
        42870100-    44518950kb: ..................................................
        44518950-    46167800kb: ..................................................
        46167800-    47816650kb: ..................................................
        47816650-    49465500kb: ..................................................
        49465500-    51114350kb: ..................................................
        51114350-    52763200kb: ..................................................
        52763200-    54412050kb: ..................................................
        54412050-    56060900kb: ..................................................
        56060900-    57709750kb: ..................................................
        57709750-    59358600kb: ..................................................
        59358600-    61007450kb: ..................................................
        61007450-    62656300kb: ..................................................
        62656300-    64305150kb: ..................................................
        64305150-    65954000kb: ..................................................
        65954000-    67602850kb: ..................................................
        67602850-    69251700kb: ..................................................
        69251700-    70900550kb: ..................................................
        70900550-    72549400kb: ..................................................
        72549400-    74198250kb: ..................................................
        74198250-    75847100kb: ..................................................
        75847100-    77495950kb: ..................................................
        77495950-    79144800kb: ..................................................
        79144800-    80793650kb: ..................................................
        80793650-    82442500kb: ..................................................
        82442500-    84091350kb: ..................................................
        84091350-    85740200kb: ..................................................
        85740200-    87389050kb: ..................................................
        87389050-    89037900kb: ..................................................
        89037900-    90686750kb: ..................................................
        90686750-    92335600kb: ..................................................
        95633300-    97282150kb: ..................................................
        97282150-    98931000kb: ..................................................
        98931000-   100579850kb: ..................................................
       100579850-   102228700kb: ..................................................
       102228700-   103877550kb: ..................................................
       103877550-   105526400kb: ..................................................
       105526400-   107175250kb: ..................................................
       107175250-   108824100kb: ..................................................
       108824100-   110472950kb: ..................................................
       110472950-   112121800kb: ..................................................
       112121800-   113770650kb: ..................................................
       113770650-   115419500kb: ..................................................
       115419500-   117068350kb: ..................................................
       117068350-   118717200kb: ..................................................
       118717200-   120366050kb: ..................................................
       120366050-   122014900kb: ..................................................
       122014900-   123663750kb: ..................................................
       123663750-   125312600kb: ..................................................
       125312600-   126961450kb: ..................................................
       126961450-   128610300kb: ..................................................
       128610300-   130259150kb: ..................................................
       130259150-   131908000kb: ..................................................
       131908000-   133556850kb: ..................................................
       133556850-   135205700kb: ....................
    san001 cli%

There were no Xs listed so our deltas are fine. This took an hour for our 130GiB Oracle database! checkrcopyvv should be used for troubleshooting purposes only for HP technical support, and not for us users, which is why I've blogged this :) Time for us to dig deeper and luckily the Storage Processor runs RedHat and the nodes run Debian.. .. Hmm.