Why Is RESNET Limiting Its Options for Improvement?
In case you haven't heard, last year RESNET created a task force to look at ways to improve the consistency of results among certified home energy raters (aka HERS raters). One week ago, executive director Steve Baden sent an email about the latest developments in the process. He also attached a document titled Quality Improvement Working Group RESNET QA Options (pdf) that outlines the areas of consensus and gives three options for changing the way quality assurance (QA) is done. Unfortunately, there's a glaring omission from the list.
The need for improvement exists
We've been a HERS rating provider, now called HERS QA provider, since 2009. We serve independent raters from Arizona to Maine, and we make sure our raters know what they're doing and do things right. Because we don't let them get away with not meeting the HERS Standards or ENERGY STAR new homes program guidelines, we hear their complaints about raters with other providers who aren't such sticklers. Here's one I got last month:
It's getting more and more frustrating on our end and is to the point where [one local HVAC company] is not wanting to bid on projects with a builder that has [our company] as the raters. Meanwhile, [that HVAC company] is pushing paperwork / installation issues through other raters that are apparently not having any issues getting Energy Star.
Yes, there are problems. Not all are egregious violations. Part of what's motivating RESNET is that if you hire several raters to come into a home to do a rating, you'll get a spread of results that's broader than it should be. Apparently, some of the big production builders that build homes in different markets have experienced this when they build the same home in a similar climate and get different results when different raters rate the home.
When a provider does QA on their raters, the results for the HERS Index are supposed to be within 3%, but different providers and QA Designees (QADs) don't always interpret the standards the same way. Each one has their ways of doing things and might emphasize or de-emphasize certain aspects of the rating process. So you can have variance in the results even without malfeasance.
RESNET's oversight of providers has improved but...
The HERS rating industry is set up to have QA of raters done by their providers. Providers are accredited by RESNET, which oversees and does QA on the providers. Energy Vanguard Energy Ratings has to submit an annual report to result documenting the results of the QA we do. In our first three years as a provider, however, RESNET wasn't doing a lot of oversight. One of those years, we never even got a report back from RESNET about how we did with our annual report. Also, despite RESNET having guidelines calling for enhanced oversight or field visits to providers, we haven't had any of that. (We're getting our first online review this fall.)
That's definitely changed, though. Laurel Elam took over as QA Manager for RESNET, and she's on the ball. In the past two years, she's not only looking carefully at the reports submitted by all providers, she's acting on what she finds. Here's what a recent email from RESNET announced about the process:
The reviews found numerous instances where providers fell short of complying with RESNET quality assurance provisions. As a result 43 Rating Quality Assurance Providers were placed on administrative probation. An additional 56 Providers had findings that needed correction to be compliant with RESNET Standards.
That's great! I'm happy to see that RESNET is starting to hold providers to the requirements in the HERS Standards.
But, it's still not enough. Laurel is doing a wonderful job and RESNET is lucky to have her, but she's not a technical person. She's never been a HERS rater and isn't a QAD. I don't think the QA Manager needs to have those qualifications, but RESNET has to have technical people involved in the oversight process. They do have at least one technical person, Abe Kruger, who assists on a contract basis, but he's not fulltime.
What RESNET has been doing over the past two years since Laurel took over is the important step of making sure the t's are crossed and the i's dotted. What RESNET has not been doing is looking at the words and seeing if there should even be a t or an i in them or maybe if they should have used a different word altogether.
In other words, RESNET needs to dive in deep enough to determine, not only if providers are doing enough QA and keeping records properly, but also if their QA process is good enough to assure the quality of ratings. A big step here would be to repeat some of the rating data file QA done by providers on raters.
Providers are required to do this for 10% of all ratings done by a rater, so RESNET, it seems to me, should be doing this for some percentage of every provider's file QA. RESNET wouldn't have to travel for this, and I think it would turn up a lot of problems and force all providers to make sure they do it right.
The 3 options
Back to the Quality Improvement Working Group now. This process has been going on since last year, and they recently released a document outlining the issues and their options for improvement (pdf). Here they are:
- RESNET staff do all QA on raters and ratings.
- RESNET creates a pool of contractors to do all QA on raters and ratings.
- RESNET contracts with a pool of third-party regional QA contractors to do all QA on raters and ratings.
I can't see RESNET going with number 1. They probably won't want to do the work involved with number 2 either. I think, if they continue down this path, they'll probably go with number 3, and we'll have ICF, CSG, and ClearResult doing all the QA.
All three of these options take QA away from the current providers. The task force and working group have said they want all QADs to act as "agents of RESNET." Providers will still exist but will not do QA anymore.
It's an important document for the future of the HERS rating industry. Please read it, think about it, discuss it, and participate in the public comment process. Unfortunately, today is the deadline for public comment, so don't delay.
The glaring omission
Let me state again that this process is necessary. RESNET definitely needs to do something improve the consistency and quality of the HERS ratings being done. The list of consensus items in their recent document has some good points. For example:
"QA contractors should not be performing QA on Raters and ratings where a conflict of interest exists."
"The RESNET Home Energy Rating and QA processes must be more clearly
defined to achieve greater consistency."
"Qualifications of the companies and/or individuals who provide QA services,
including accountability, must be more stringent and more clearly defined to
create greater consistency and quality in how QA is delivered."
Unfortunately, they seem to think that because the majority of providers also do ratings, no providers should be doing QA. I beg to differ. There are a few of us who are providers but do little or no ratings. Energy Vanguard does a handful of ratings per year, nearly all for Habitat for Humanity, many of them pro bono. We made the decision early on not to compete against our raters. Building Efficiency Resources (The BER) is another company with this model, and so is Southface.
Why is RESNET ignoring the benefits of this model? Rather than including only options that take QA away from providers, why not have a fourth option that allows providers to do QA but not on any ratings they do themselves? This is the fox-in-the-henhouse issue people have been talking about for years because of all the big providers that do their own QA.
This model can work, but it requires RESNET to take the next step I described above and look not only at providers' administrative compliance but also technical compliance. As a member of the RESNET QA committee, I've been saying this over and over. The current structure can work but RESNET needs to do more thorough QA on providers.
What happens when QA is taken away from providers?
If RESNET continues down the current path and chooses one of the three options above (probably the third), why would anyone want to be a provider? The big rating companies will continue, of course, but there's not much in it for small providers like ours. We'll become document chasers and discipline enforcers, and I have no interest in that.
If RESNET chooses one of the three options above, I think QA could actually get worse for independent raters. Many providers do more than the minimum QA. In our providership, we look at nearly every REM/Rate file that gets sent to us, not just the 10% required by RESNET. Will the new QA companies do that?
And providers won't know as much about QA if they're not doing it. When raters call them up to ask questions, providers won't know as many of the answers. They may even steer raters in the wrong direction because the QA companies may do things differently.
The collapse of QA
Another problem with the three options above is that it could lead to the collapse of the QA process from three levels to two. Right now it's RESNET => providers => raters. It seems like RESNET thinks that handing all the QA off to "independent" QA companies (or RESNET staff) will solve all the problems so the collapse won't matter. I think they're wrong. Let's keep three levels and beef up RESNET's QA, which will reverberate all the way down.
I don't think the folks on the task force and the working group have fully considered all the options or the consequences.
Two changes that could make all the difference
I hope you'll comment on this issue before the deadline (tonight, 8/29). If you want to comment but can't spend a lot of time, you could simply tell RESNET to consider the following two changes:
- Providers should not be allowed to do QA on their own ratings.
- RESNET should do more technical QA on providers.
The latter is what I was talking about above when I said RESNET should redo some percentage of each provider's rating data file QA. Combined with the great job Laurel Elam has been doing with administrative QA, this could would allow RESNET to discover problems that might otherwise never be found.
Let's get it right
If you're a home energy rater, QAD, or provider, please get involved in this process. Read the documents. Check out the resources I've listed below. And most of all, participate in the public comment process. It ends today.
Submit a comment to RESNET — Deadline 8/29!
NOTE: Comments are moderated. Your comment will not appear below until approved.