BrianGreenberg.net — Fractional CIO - How do you report on your backup utilization?

For all of you doing heavy reporting on NetBackup (NBU), I’ve discovered an inconsistency about how NBU reports job information between bpdbjobs and bpimagelist. If you’re charging users for backup services, this can present incorrect billing information.

I discovered quite a while ago, that what you are reporting on for KB backed up may in fact be incorrect. This is going to depend entirely on your reporting tools as well as if you are backing anything up via NDMP.

Here’s the scenario, and keep in mind, this pertains to NDMP based backups, not regular client backups (e.g. Windows, *nix, etc.). Many, if not all, backup reporting applications get their data on utilization from bpdbjobs. The problem with this is bpdbjobs and bpimagelist do not always report the same numbers on number of files backed up nor KB backed up. The numbers are almost always different for NDMP backups and the bpdbjobs numbers are almost always lower. The authoritative number that Symantec considers final is the number that comes from bpimagelist and a few tests have confirmed that for me.

The following is a sample of what I can see in my environment. Note the NDMP v. NON-NDMP jobs by the “*” in the NDMP column below.

In the following columns, FILE and KB DIFF are always the data of bpdbjobs – bpimagelist for the respective fields. (PasteBin link)

JOB ID              IMAGE ID              JOB KB         IMAGE KB    |    JOB FILES      IMAGE FILES   | FILE DIFF  KB DIFF  NDMP
------- ---------------------------- --------------- --------------- | --------------- --------------- | --------- --------- ----
2502003 abcdefghi02_1217447043            72,563,968      72,563,971 |              11              13 |        -2        -3   *
2501975 abcdefghi01_1217432409             1,615,872       1,615,872 |               6               6 |         0         0
2501974 abcdefghi01_1217432405            20,480,256      20,480,256 |              17              17 |         0         0
2501973 abcdefghi01_1217432453            40,405,984      40,405,984 |              13              13 |         0         0
2501972 abcdefghi01_1217432408             2,610,080       2,610,080 |              34              34 |         0         0
2501971 abcdefghi01_1217432407             1,748,064       1,748,064 |              34              34 |         0         0
2501970 abcdefghi01_1217432406             3,935,872       3,935,872 |              40              40 |         0         0
2501969 abcdefghi01_1217432404             5,286,752       5,286,752 |              68              68 |         0         0
2501968 abcdefghi01_1217432403                 1,344           1,344 |               4               4 |         0         0
2501965 abcdefghi02_1217432164           761,319,680     761,321,861 |          17,032          17,019 |        13     -2181   *
2501964 abcdefghi01_1217430086                 8,224           8,224 |               2               2 |         0         0
2501963 abcdefghi01_1217430085             3,680,864       3,680,864 |               8               8 |         0         0
2501962 abcdefghi01_1217430084                 8,224           8,224 |               2               2 |         0         0
2501961 abcdefghi01_1217430083                 8,224           8,224 |               2               2 |         0         0
2501960 abcdefghi01_1217430081             5,780,288       5,780,288 |             174             174 |         0         0
2501959 abcdefghi03_1217430001                     0               3 |               7               6 |         1        -3   *
2501958 abcdefghi01_1217430033                    32              32 |               0               0 |         0         0
2501954 abcdefghi02_1217422677         1,268,344,320   1,268,347,440 |          22,093          43,631 |    -21538     -3120   *
2501940 abcdefghi02_1217420903         1,149,116,672   1,149,117,882 |           8,061          16,069 |     -8008     -1210   *
2501924 abcdefghi02_1217419446           897,271,296     897,272,696 |           9,603           9,554 |        49     -1400   *
2501922 abcdefghi02_1217419345           212,359,424     212,382,178 |         188,001         374,995 |   -186994    -22754   *
2501921 abcdefghijk01_1217415601         159,289,854     159,289,854 |             150             150 |         0         0
2501920 abcdefghi04_1217412000            11,941,888      12,004,384 |       1,011,153       1,394,050 |   -382897    -62496   *
2501919 abcdefghi05_1217411094             7,997,184       7,997,184 |           9,013           9,013 |         0         0
2501912 abcdefghi06_1217408402             1,884,160       1,885,176 |          29,501          33,791 |     -4290     -1016   *
2501911 abcdefghi06_1217408401                   256             292 |           1,443             989 |       454       -36   *
2501907 abcdefghi06_1217406600           118,403,584     118,403,618 |             329             501 |      -172       -34   *

Now for many of you this is not a significant issue. However, if you are charging back to your businesses/customers on KB backed up as in a typical utility model for services provided you may be loosing a lot of money.

I have tested CCS/BR from Veritas/Symantec, Aptare, WysDM/EBA-EMC and in every case I was getting bpdbjobs numbers where the overall KB backed up was far less than what was actually being backed up. In my environment, which is 80% NDMP based, this came out to be a significant figure. A couple notes about the reporting products I used; the version of CCS is a couple years old and the folks at Aptare worked closely with me in recognizing the problem and have aledgedly fixed the problem by comparing all completed jobs from bpdbjobs to that of bpimagelist and then using the bpimagelist numbers as the authorative numbers in their database but I haven’t had the opportunity to test that.

I’m told by Symantec that the reasons for the discrepencies between bpdbjobs and bpimagelist numbers for NDMP based backups seems to be in the fact that NDMP is doing block based calculations and providing block sizes back to bpdbjobs and somehow bpimagelist is getting correct file based calculations when the job is complete.

Regardless as to why this occurs, the fact of the matter is that the numbers are different and they shouldn’t be, especially when considering that most organizations and service providers are charging back and/or generating revenue from the data that most reporting engines seems to be collecting from bpdbjobs and not bpimagelist.

I still have a case open with Symantec.

Update: An update from Symantec. Apparently the current version of CCS allows you to collect from bpdbjobs and bpimagelist and choose which to use for your utilization reporting.

One response to “How do you report on your backup utilization?”

Stephen Foskett

January 4, 2009

Excellent post, Brian! Nice to have you contributing your $0.02 in the storage sphere!

I ran across this in the past myself. I had to reconfigure some perl-based backup reporting scripts to use bpimagelist because it was much more reliable.