RESIDENTIAL ENERGY CONSUMPTION SURVEY QUALITY PROFILE

5. Measurement Error

This chapter presents information about measurement errors associated with the data collection

phase of RECS. Measurement error can contribute to the total error of survey estimates in the

form of bias or nonsampling variance. Direct or indirect quantitative information about

measurement errors in RECS can be obtained in several different ways.

Special data collection procedures, which generally cost more than standard interviews but are

believed to provide more precise information, can be used to collect information for selected

households. These procedures often involve various kinds of direct observation and physical

measurement, as opposed to merely asking for information from survey respondents. Such

procedures include energy assessments in which data on household characteristics are collected

by trained technicians. Other examples are the collection of nameplate data in order to obtain

more precise information on the characteristics of central air-conditioning units and the collection

of information on thermostat settings and temperatures by direct observation rather than by asking

respondents to report them. Another useful procedure has been to conduct personal or telephone

interviews with "outliers"--i.e., households identified in the data processing phase of the survey

as having reported unusual or apparently inconsistent values for selected items. These kinds of

procedures are reviewed in the first section.

Comparisons of data for the same household from different sources provide another kind of

information about measurement error. Cross-sectional comparisons involve data for a household

for the same time period from the household, rental agent, and supplier surveys; longitudinal

comparisons involve data for the same household from successive survey years. Weather data

assigned to households in a specified geographic area can also come from more than one source.

Results from these types of comparisons are presented in the second section.

The level of measurement error can also be affected by the design and format of the survey

questionnaire and by the type of training administered to interviewers. Information on these

topics is presented in the final section.

RECS estimates of end-use energy consumption within households are obtained indirectly from

survey data by allocating total consumption to various uses on the basis of a nonlinear regression

model. These estimates and their evaluation through submetering studies are covered in Chapter

7, "Estimation and Sampling Error." Macro-comparisons, that is, comparisons of RECS estimates

with comparable data from other surveys conducted by EIA and from surveys conducted by other

agencies and organizations, are discussed in Chapter 8, "Comparisons of RECS Estimates with

Other Data."

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Special Data Collection Procedures

Energy Assessments

In 1979, following the National Interim Energy Consumption Survey (NIECS), an Energy

Assessment was undertaken by Technology and Economics, Inc., of Cambridge, Massachusetts,

in a subsample of 44 of the NIECS sample households. Trained technicians visited the

households, all but two of which were single-family households. They measured floor areas,

counted windows, examined insulation and noted the characteristics of space-conditioning

equipment and selected appliances. Their observations were compared with the responses to the

NIECS interviews (Blumstein, York, and Kemp 1981).

This Energy Assessment was undertaken as a pilot test for a continuing program of assessments

that was being considered as a regular part of RECS (Response Analysis Corporation 1980,

Part 6). The plan was to perform such assessments for a subsample of the households included

in each Household Survey. However, resources available for RECS proved to be insufficient to

implement this plan and there have been no further assessments of this type. The generality of

the Assessment findings was limited by the use of a small convenience sample, lack of fully

standardized procedures, and limited training for the technicians. In addition, the data collection

instrument was not designed for direct comparisons with corresponding NIECS data items and

there was no followup to reconcile differences between the two sources of information.

Nevertheless, the Assessment provided useful information about possible sources of measurement

errors in NIECS and subsequent surveys.

There were large differences between the NIECS and Assessment data on square feet of

floorspace (Table 5.1). For 14 of the 27 households that had usable measurements from both

sources, differences were 25 percent or more of the Assessment values. NIECS respondents had

been asked to give their best estimates of floorspace; in the Assessment the technicians made

measurements. Some of the discrepancies may have been due to a conceptual difference: NIECS

respondents were asked to report square feet of living space, while the Assessment technicians

were asked to measure "conditioned space," including only rooms and other enclosed areas with

some direct means of heating. On this basis, one might expect the NIECS values to be somewhat

larger; nevertheless, for 9 of the 27 households, the NIECS values were 25 percent or more

below the Assessment measures.

Despite their limitations, the Assessment findings on floorspace demonstrated that respondent

estimates of floorspace were likely to be subject to unacceptably large errors. Consequently,

from RECS survey year 1980 on, measurement by survey interviewers has been the preferred

method of obtaining information on floorspace. The measurement procedure used in the

Assessment was itself not fully satisfactory and has been replaced in RECS by procedures that

are believed to be easier to use and more reliable.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.1. Comparison of NIECS and Energy Assessment Data on Floorspace

Percent

Difference

(NIECS - EA

100)

Number of Households

NIECS > EA NIECS < EA Total

0to9.9

10.0 to 24.9

25.0 to 49.9

50.0 and over

Total 15 12 27

There was no space information from one or both sources for 17 of the EA households.

Source: Blumstein, York, and Kemp,

An Assessment of the National Interim Energy Consumption Survey

(1981), Table 8.

Some other findings from the comparison of NIECS and Energy Assessment data were:

• There were many differences between counts of windows, both by type and overall

for the household.

• The presence or absence of attic insulation was reported accurately, but there were

substantial differences in reports of the thickness of insulation used.

• Reports of fuel used for heating and other purposes were generally in agreement. An

exception was fuel used for dryers; of 33 households for which the dryer fuel was

reported in both NIECS and the Assessment, there were differences for 7, all of which

reported electricity in the Assessment and gas in NIECS.

• Several differences were observed in the numbers of refrigerators and separate food

freezers reported and in the characteristics of refrigerators, such as temperature

controls and automatic defrost/frost free features.

These findings from comparisons of NIECS and Energy Assessment Data were taken into account

in the determination of content and formulation of questions for subsequent surveys.

Collection of Nameplate Data

In the 1990 Household Survey, interviewers were asked, for single family houses with central

air-conditioning, to record manufacturer’s name, model number, year manufactured, and other

information from the nameplate of the outside unit (Hall 1992). The main purpose of collecting

this information was to obtain a measure of rated efficiency for each housing unit’s central air-

conditioning equipment. This was to be done by matching the make and the model year and

number against semi-annual directories of equipment characteristics issued by the

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Air-conditioning and Refrigeration Institute (ARI). For all successful matches, the seasonal

energy efficiency ratio (SEER) for the equipment was entered into the RECS data file.

Table 5.2 shows the results of the attempts to acquire SEER’s for central air-conditioning

equipment. No nameplate data were obtained for 26.5 percent of the 1,820 households with

central air-conditioning, either because they were located in multiunit buildings or because they

had responded by mail. Directory matches were attempted for the remaining 1,337 households;

SEER’s were obtained for only 24.8 percent of those households, or 18.2 percent of all sample

households with central air conditioning. The most frequent reasons for failure to find a SEER

in the directories were failure to match on manufacturer’s name and failure to match on model

number.

A subsequent effort was made, for the 331 households for which SEER’s had been obtained, to

obtain capacity values from the ARI directories. Values were located for 279 (84.3 percent) of

these households. In view of the high cost and limited success of the nameplate data collection

and matching operations in the 1990 RECS, they were not undertaken in 1993.

Table 5.2. Results of Matching Nameplate Data Against ARI Directories to Obtain Seasonal Energy

Efficiency Ratios: 1990 RECS

Outcome

of Match

Number

of Units

Percent

Of Total

Of Attempted

Matches

Households with central air conditioning

No match attempted

Mail questionnaire

Multi-unit building

Match attempted

Successful, SEER obtained

No SEER obtained

No match on make

No model year

No match on model

No SEER available

1,820

483

384

1,337

331

1,006

574

283

126

100.0

26.5

5.4

21.1

73.5

18.2

55.3

31.5

1.3

15.6

6.9

100.0

24.8

75.2

42.9

1.7

21.2

9.4

Unit manufactured prior to 1980 or no SEER in directory.

Source: Hall,

Nameplate Data Collection in the 1990 RECS

(1992).

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Checking Thermostat Readings

Since 1981, the RECS Household Survey questionnaires have included questions on average

temperatures maintained in the home in the wintertime under three conditions: during daytime

when someone is home, during daytime with no one home, and during sleeping hours. If

respondents say they cannot report temperatures but can give thermostat settings, the latter are

accepted. These self-reported temperatures are characterized in the survey reports as follows:

The self-reported temperatures, especially for some respondents, are impressions of

typical temperatures and may not represent actual temperatures, or the averages of actual

temperatures in the home. (EIA 1993a, p. 148)

There have been no attempts in RECS to collect information about indoor temperatures or

thermostat settings by direct observation. However, a study in a small city in New York State

provided some information on the accuracy of self-reported thermostat settings (Luyben 1982).

Data were collected for one sample of households by personal interviews and for another sample

by telephone. In the telephone survey, respondents were first asked to report their thermostat

settings and then to go to their thermostats and check the reported values. The mean of the

checked values was significantly higher than the reported values by 0.6 degrees Fahrenheit.

In the personal interviews, the interviewers recorded observed values of thermostat settings and

temperature readings. Temperature readings exceeded thermostat settings by a mean of 0.8

degrees. The mean of the observed settings was significantly higher than the mean of the

checked settings from the telephone survey households: 68.3 versus 67.0 degrees.

Detection and Evaluation of Outliers

The detection and analysis of outliers can be a useful technique for understanding survey

responses, identifying and controlling survey errors, and improving survey processes. Outliers

are reported values that lie at the extremes of a univariate or multivariate distribution of variables

included in the survey. The analysis of outliers can include recontacts with survey respondents

to determine whether there were errors in the values initially reported and whether there were

special circumstances to explain the unusual observations.

In March of 1984, in-depth reinterviews were conducted with eight households that had

participated in the 1981 RECS and for which data on consumption were available from suppliers

(EIA 1984b, Appendix G, Erickson 1984). The method used to identify these eight households

as outliers was to impute their consumption of specific fuels, using the regression models

normally used to impute missing data on consumption and to compare the imputed values of

consumption with the values actually provided by the suppliers. A purposive sample of eight

households showing large differences in either direction for consumption of electricity, natural

gas, or fuel oil was selected for the interviews. These were households whose consumption

appeared to be far out of line with what might have been expected on the basis of housing unit

characteristics and household behaviors that had been reported in the initial interviews.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

There were two sets of four interviews each. Set A was conducted by a pair of interviewers, and

Set B by a single interviewer. All used conventional ethnographic interviewing techniques.

Their primary goal was to arrive at an explanation for the unusually high or low consumption,

but they also investigated several broad topics, including family interactions, recreation, home

improvements and attitudes toward utility companies, nuclear power, conservation, rising costs

of energy, and family finances.

The main findings of the eight in-depth interviews are presented in Figure 5.1. The interviewers

were successful, for the most part, in finding reasonable explanations for the extreme values in

consumption of specific fuels. The explanations proved to be more or less equally divided

between reporting errors in the initial RECS interviews (six of the eight households) and unusual

circumstances affecting consumption, some of which may not have been fully reflected in the

imputation model (also present in six of the eight households).

An important motivation for undertaking these interviews was to determine whether the

questionnaire for the 1984 RECS could be expanded to include information that would help to

explain patterns of unusually high or low consumption. After the findings were reviewed, no

questions were added but consideration was given to other changes in survey procedures, such

as improved interviewer training, additional processing steps, and followup interviews. One

outcome has been the inclusion of "model-based outlier checks" as a standard part of data

processing. In processing the 1990 RECS data, for example, there was a manual review,

sometimes involving telephone calls to respondents, of data for all households for which the

model-based estimate of fuel consumption was more than three times or less than one-third of

the value based on Supplier Survey data (Response Analysis Corporation 1992b, p.7-14).

Another outlier investigation associated with the 1981 RECS had to do with data on temperature

settings, a topic that had been included in the questionnaire for the first time in that survey year

(Thompson 1982, Day 1982). The survey contractor made telephone calls to 9 respondents who

reported maintaining (with a thermostat, radiator valve, or other control) nighttime temperatures

higher than their daytime temperatures (presumably a reversal from normal behavior) and to 9

respondents who reported nighttime temperatures substantially lower than their daytime

temperatures.

In the first group, nighttime higher than daytime, eight of the nine respondents called changed

their responses in ways that reduced the differences; however, all but two of the group

confirmed that they purposely maintained higher temperatures at night and provided explanations

for that behavior. In the second group, nighttime much lower than daytime, all but one

confirmed their original responses, although some said they were uncertain about the precise

temperature levels. Most of the explanations involved use of electric blankets or a warm

combination of non-electric blankets. The findings suggested that responses to questions about

temperatures maintained in the housing unit are subject to sizable response errors.

An analysis of outliers in the 1984 RECS by Latta (1988) suggests the potential power of this

method. The goal of Latta’s analysis was to improve the nonlinear regression model used to

impute missing entries for heated floorspace. He used data for sample housing units with

complete information to estimate the parameters of the proposed model and observed that there

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

were several extreme outliers, units with large differences between reported and imputed values.

An examination of the data listings for these units showed substantial clustering by PSU and

interviewer, leading to a hypothesis that a few interviewers may have been making systematic

errors in identifying which portion of total floorspace was heated. One set of outliers consisted

of four townhouse units, each with three floors whose dimensions had been recorded as 7 by 30

feet, a rather unlikely set of measurements. Because this analysis was undertaken well after

completion of the 1984 RECS, there was no followup on these particular cases. However, the

findings indicate that comparison of reported and imputed values followed by review and

followup of outliers is a promising technique for quality improvement.

Comparisons of Individual Household Data from Alternate Sources

The design of RECS provides built-in opportunities to investigate the nature and size of

measurement errors through the analysis of multiple observations of the characteristics of

individual housing units or households. The longitudinal component of the sample design

provides observations for the same housing units at different times. The collection of overlapping

data for selected items from three sources--households, rental agents and suppliers--provides

duplicate observations for the same housing units at the same time or covering the same time

period. The interpretation of data from multiple observations is not necessarily straightforward;

the sources of the observations must be carefully considered to decide what they tell us about the

effects of response bias or response variability on the survey estimates.

Longitudinal Comparisons

As described in Chapter 2, the RECS samples for 1982, 1984, 1987, and 1990 each contained

a subsample of housing units which had been included in the sample in the preceding survey

year. Because a large proportion of the questionnaire content is repeated in successive survey

years, responses to comparable items for the same unit in the 2 years can be compared. The

interpretation of observed differences is not obvious. For some housing unit characteristics, such

as year built and type of housing unit (mobile home, single-family detached, etc.), there should

be no differences from one survey year to another, so that differences are almost certainly due

to errors in data collection or processing. For other housing characteristics, such as appliances,

types of fuels used, and even number of rooms, real changes can occur. Real changes in

household characteristics, like number of persons and family income, can occur whether or not

a different household occupies the housing unit in successive survey years.

Table 5.3 provides information about differences in selected items for housing units that were

occupied by the same households in the 1980 and 1982 RECS (Thompson 1985a). In an effort

to determine the reasons for individual differences, telephone calls were made to households for

which the responses for 1980 and 1982 differed for one or more of the selected items. Only 71

percent of the differences were checked in this way: 12 percent were eliminated to reduce

burden on households with more than three items showing differences and 19 percent were

associated with households that could not be reached by telephone. Thus, the final column of

the table, showing the percent of differences "unexplained," includes some differences for which

interviewers and respondents could not provide any explanation and some which were not

covered by telephone calls to respondents.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.3. Differences Between Responses Reported by the Same Household in the 1980 and 1982

RECS

Differences Percent of differences:

Item Base

Number %

Explained

Real

Change

Explained

Errors

Unexplained

1. Number of windows

2. Year the house was built

3. Main home heating

equipment

4. Number of stories

5. Year moved in

6. A/C equipment present

7. Number of rooms

8. Use a home freezer

9. Basement heated or

unheated

10. Number of refrigerators

11. Full basement/part

basement

12. Availability of natural gas

13. Use a clothes dryer

14. Type of living quarters

15. Number of bathrooms

16. Main home heating fuel

17. Main water heating fuel

18. Presence of a basement

19. Main cooking fuel

1,398

1,296

1,394

999

1,397

1,398

1,395

1,111

1,381

1,111

370

1,397

1,400

1,387

1,398

1,393

1,111

1,400

337

296

206

145

194

179

165

149

110

128

115

107

24.1

22.8

14.8

14.5

13.9

12.8

11.8

10.7

9.9

9.3

8.6

8.2

7.6

6.6

6.2

6.0

4.1

2.9

Base excludes households for which 1980 RECS response is imputed or unknown and those for which 1982 RECS response is

unknown.

Some responses are grouped for these items. For a difference to be counted, it must be >3 windows; the years 1975-79 were

combined into one category; hot water pipes and radiators were combined as one heating system; full and 1/2 baths were combined

and each counted as one bathroom; the difference between number of rooms must be >1.

Single-family homes.

Single-family or mobile homes that do not use natural gas.

Errors by respondents, interviewers, coders, and data entry operators.

Source: Thompson,

Utility of Paying Respondents: Evidence from the RECS

(1985).

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

For 9 of the 19 items shown in Table 5.3, real change accounted for at least 10 percent of the

observed differences. These are minimum values, because some of the unexplained differences

may have resulted from real changes. Such real changes appear to be largely related to the

acquisition of new appliances and heating or cooling equipment and to changes in the availability

of natural gas, making possible changes in the main heating fuel used.

For 6 of the 19 items, the attempt at reconciliation of differences confirmed that at least half of

them resulted from errors in data collection or processing. These were items for which one

would expect few, if any, real changes to occur: number of windows, number of stories, type

of living quarters, presence of a basement, basement heated or unheated, and full basement or

part basement. The general conclusion to be drawn from these findings is that, at the level of

the individual housing unit, real changes over time are difficult to distinguish from differences

due to measurement errors. Because essentially the same data collection procedures were used

in both years, we can also conclude that estimates for some housing unit characteristics, notably

number of windows, are subject to high response variability.

Measures of total and heated floorspace were also compared for a subsample of 355 housing

units included in the 1980 and 1982 RECS (EIA 1984b, p. 114-115). The results for 300 housing

units that had usable square footage data for both years are shown in Table 5.4. Averages for

the total and single-family detached units were fairly close for the two survey years. However,

the median absolute percent differences between values for individual units for the two years

were relatively large, 11.7 percent overall for total square footage. They were larger, at 15.6

percent, for heated square footage, probably because of uncertainties about the interpretation of

the concept of a "heated area," possibly also in part because of some real changes in this item.

Longitudinal comparisons of 1982 and 1984 data, also with telephone calls to explain differences,

were undertaken following the 1984 RECS. In this instance, only seven topics were selected for

analysis: main home heating fuel, main water heating fuel, air- conditioning equipment and fuel,

clothes dryers, home freezers, dishwashers, and availability of natural gas. Telephone contacts

were successfully completed for 505 (76 percent) of the 668 differences that were found for these

seven topics. Real changes explained 42 percent of these differences; virtually all of the rest

resulted from errors in the 1982 or 1984 values or, in a few instances, errors in both years (EIA

1987d).

Records for the longitudinal differences for which respondents were successfully contacted are

included in the public use files for the 1982 and 1984 RECS. There is a separate record for each

difference showing the topic number, a code for the interpretation of the difference (year 1

correct, year 2 correct, neither year correct, real change, or cannot determine) and a code

identifying the reason for the error, if one occurred. As noted below, some longitudinal

comparisons have been made for 1984-1987, and comparisons are possible for 1987-1990, but

no followup contacts were made, following the 1987 or 1990 RECS, to determine reasons for

differences.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.4. Differences in Square Footage Reported for the Same Household in the 1980 and 1982

RECS

Housing Type

Item Total

Single-Family

Detached

Mobile

Home

Multi-unit

Building

Housing

Type

Responses

Differ in

1980 & 1982

Number of cases

Average Square Feet

per Housing Unit

1980

1982

Median Percent

Difference in Square

Footage

Average Heated Square

Footage per Housing

Unit

1980

1982

Median Percent

Difference in Heated

Square Footage

300

1,797

1,821

11.7

1,536

1,521

15.6

208

2,116

2,142

11.8

1,780

1,751

16.9

803

721

7.2

798

711

7.2

1,082

1,147

12.2

966

1,039

14.4

1,503

1,282

11.3

1,469

1,194

13.4

Units that had good square footage data for both years.

Source: Energy Information Administration,

Consumption and Expenditures

(1982).

Table 5.5 shows results of a comparison of 1984 and 1987 RECS data on type of housing unit

for units that were included in both surveys. The final column shows the index of inconsistency

for each category. The index of inconsistency is a measure of the percent of total variance for

an item that is accounted for by response variance. As a rough rule of thumb, response variance

is considered to be low when the index is less than 20, moderate for values between 20 and 50,

and high when it is greater than 50. (For further discussion and a formula for calculating the

index, see Groves (1989).) The value of the index for the "single family attached" category is

at the upper end of the moderate range, indicating that there were frequent difficulties in

distinguishing such units from single family detached units and from those in apartment buildings

with two to four units.

Table 5.6 shows a comparison, also based on housing units included in both the 1984 and 1987

RECS, for reports on year of construction of the housing unit (Battles 1991a). This tabulation

is limited to housing units that were occupied by the same household in 1984 and 1987 and for

which the householder was the respondent in both years. The values of the index of

inconsistency are in the moderate to high ranges, higher for the most part than the values

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

observed for type of housing unit in Table 5.5. As one might expect, the values are smaller for

the most recent periods. They are also smaller for units built prior to 1940, presumably because

the time period covered by that category is much longer.

Table 5.5. Longitudinal Households

: Housing Type Reported in 1984 and 1987 RECS

Housing

Type

Reported

in 1984

Housing Type Reported in 1987

Index

Inconsistency

Mobile

Home

Single-

Family

Detached

Single-

Family

Attached

Apartment Bldg.

2-4 Units 5+ Units

Mobile Home

115

9 0 0 0 7.7

Single-Family

Detached

1,265

16 20 1 9.5

Single-Family

Attached

026

14 2 46.7

Apt. Bldg. 2-4

Units

01021

209

10 19.2

Apt. Bldg. 5+

Units

0 0 6 10

269

5.9

Tabulation excludes 15 cases where it was determined that different housing units had been interviewed and one case where the

basement had been converted to an apartment.

Source: Energy Information Administration,

Housing Characteristics

(1987).

Table 5.6. Longitudinal Households

: Year of Construction Reported in 1984 and 1987 RECS

Year of

Construction

Reported in

1984

Year of Construction Reported in the 1987 RECS

Total

Units

Index of

Inconsis-

tency

Before

1940

1940-

1949

1950-

1959

1960-

1969

1970-

1974

1975-

1979

1980-

1984

Before 1940

333

42 26 22 12 6 1 442 30.4

1940 to 1949 27

19 7 2 0 2 116 57.7

1950 to 1959 23 15

134

34 7 4 2 221 49.3

1960 to 1969 9 11 40

145

25 13 4 247 47.8

1970 to 1974 8 6 6 22

17 7 161 48.2

1975 to 1979 305326

114

9 160 32.7

1980 to 1984 104137

68 29.8

Total Units 404 133 236 234 170 161 77 1,415

Housing units occupied by the same household in 1984 and 1987.

Source: Battles,

Effects of the Adjustment of 1990 Census Data on the 1990 RECS Control Totals Obtained from the Current Population Survey

(December 1991).

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Cross-Sectional Comparisons

As a result of the multi-stage clustered sample design used in RECS, the sample of housing units

sometime contains two or more units from the same multiunit structure. When this occurs, there

are some housing unit characteristics, such as the year in which the structure was built and the

main space and hot water heating fuels, that one would expect to be the same for every unit in

the structure. When differences are found for these characteristics in "inter-case comparisons"

of different units in the same structure, they can be taken as indications of response error.

Blumstein, York, and Kemp (1981) used data from NIECS (the 1978 RECS) to make such inter-

case comparisons. There were some difficulties in determining which sample housing units came

from the same structure, but by matching on structure characteristics and identifiers, the

investigators succeeded in identifying 78 structures with more than one sample housing unit.

These structures included 305 sample housing units for which interviews were completed. When

different responses were found on items, such as year built for housing units in the same

structure, the most frequent response was assumed to be the correct one. On this basis, and

leaving out responses of "don’t know" and those that were missing, the apparent error rates were:

Number of Responses Percent

Other Than Apparently

Item Total Most Frequent Incorrect

Year Built 274 44 16

Main Heating Fuel 300 15 5

Main Water Heating Fuel 300 24 8

The 16-percent gross error rate for year built is much lower than the rate of 42 percent that can

be derived for the data on year built shown in Table 5.6. The data in Table 5.6 were based on

a longitudinal rather than a cross-sectional comparison and included all types of housing units,

not just those in multiunit structures.

As described in earlier chapters, for housing units in multiunit structures for which one or more

fuels are included in the rent, the Rental Agent Survey provides data for selected items that can

be compared with data for the same items from the Household Survey. As part of regular data

processing operations, the two sets of data are compared. When there are differences, the

response considered more likely to be correct is accepted. Except for supplemental heating fuels,

this is normally the response given by the rental agent.

Table 5.7 summarizes changes made on the basis of responses from the Rental Agent Surveys

for survey years 1981 through 1987. These data suggest what error rates for these items might

have been if the Rental Agent Surveys had not been conducted and the Household Survey

responses had been accepted. They also provide an indication of the level of error for these

items for housing units that were eligible for the Rental Agent Survey but for which no

information was obtained in that survey. The levels of nonresponse to the Rental Agent Surveys

were shown in Chapter 4, Table 4.6.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.7. Changes Resulting from Comparison of Rental Agent and Household Survey Responses:

1980-1987

Item

Survey Year

1980 1981 1982 1984 1987

Main Heating Fuel

Number of Changes

Percent of Base

Main Heating Equipment

Number of Changes

Percent of Base

Supplemental Heating Fuel

Number of Changes

Percent of Base

Water-Heating Fuel

Number of Changes

Percent of Base

Air-Conditioning Fuel

Number of Changes

Percent of Base

All Items

Number of Units in Rental

Agent Survey

Percent with >1 Changes

551

15.8

14.1

4.9

21.1

16.7

466

30.0

12.2

15.7

2.0

13.2

4.5

308

26.0

14.7

13.3

8.0

103

19.4

11.8

549

32.4

9.2

206

30.7

4.3

120

14.8

39.6

856

41.8

Base for the first 3 items in the number of units whose rental agents paid for the main heating fuel. For the fourth and fifth items,

it is the number whose agents paid for the fuel in question.

NA = Not Applicable.

Source: Energy Information Administration,

Housing Characteristics

(for the years shown).

With two striking exceptions, the proportions of eligible units (those included in the base for each

item) whose Household Survey responses were changed based on Rental Agent Survey responses

were relatively stable over the years shown. The proportions were lowest for supplemental

heating fuel because the household respondent is usually considered to be more knowledgeable

for that item. The exceptional cases were main heating equipment and air-conditioning fuel, for

which the proportion of changes in the 1987 RECS was substantially greater than in any prior

survey year. These changes may have resulted, at least in part, from changes in the questions

relating to these two items on the Rental Agent Survey questionnaires. On the 1984 Rental

Agent questionnaire there was a single item for main heating equipment, listing 13 possible

alternatives. On the 1987 questionnaire, two separate lists of heating equipment were provided,

one for units using electricity as the main heating fuel and one for units using any other fuel.

For main central air-conditioning fuel, there was a minor change: the response categories on the

1984 questionnaire were electricity, gas from underground pipes, and LPG, in that order, whereas

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

on the 1987 questionnaire the order was changed to gas from underground pipes, LPG, and

electricity. There were no changes for these items on the Household Survey questionnaires.

For the most part, there has been little overlap between the data items collected in the Household

and Supplier Surveys. For delivered fuels that are paid for directly by the household,

consumption and cost data are collected from the suppliers. For fuels whose costs are included

in rent payments, consumption and cost are imputed on the basis of housing unit and household

characteristics; this is also done when a sample household is eligible for the Supplier Survey for

one or more fuels but a response cannot be obtained from the supplier(s). To evaluate these

imputation procedures, some data on consumption for whole buildings containing sample rental

units were collected in the 1981 RECS. The findings from that study are described in the section

on "Imputation" in Chapter 6.

Following the 1993 RECS, responses to new Household Survey questions about availability of

and participation in demand-side management (DSM) programs were evaluated by comparing

them with responses to similar questions that had been included in the Supplier Survey. Several

kinds of DSM programs are offered by utilities to encourage customers to modify their patterns

of energy use, the goals being to reduce overall demand or shift some uses away from peak load

periods. The comparisons showed substantial Household Survey underreporting of the

availability of DSM programs. Of the households interviewed, 36.1 percent reported that at least

one type of DSM program was offered to them by their electric utility, natural gas utility or some

other group. By contrast, 80.6 percent of the suppliers providing electricity to the same

households reported that they offered some type of DSM program. The proportion of households

actually participating in electric or natural gas DSM programs was much smaller, but again there

was considerable disagreement between the response to the Household and Supplier Surveys.

There were many differences in both directions, but the net result was that participation appears

to have been overreported in the Household Survey (EIA 1995d, pp. 152-153).

Alternative Sources of Weather Data

The data record developed for each RECS sample housing unit includes data on weather

conditions in the vicinity of the housing unit. Of particular importance are data on heating and

cooling degree-days, both for the survey reference year and for a recent 30-year period (current

and normal degree-days). Such data have several important uses:

• When supplier data are not obtained for a housing unit for one or more fuels, the data

on heating and cooling degree-days are important inputs in the models used to impute

consumption of those fuels.

• For all housing units, the data are used as inputs to the models used to estimate end-

use consumption.

• In longitudinal analyses, variations in degree-days and departures from normal are

important determinants of variations in consumption.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Ideally, degree-day data for each housing unit would be obtained by measurement of temperatures

at the site of the housing unit. Because this is not practical, a reasonable alternative is to use

data from the more than 4,000 individual weather stations maintained by the National Oceanic

and Atmospheric Administration (NOAA). Two basic methods of using these data are possible:

(1) For each housing unit, use data from the individual weather station that is closest, in some

sense, to that unit, or (2) Use average data based on all stations in the NOAA weather division

in which the housing unit is located. NOAA has divided each of the 48 contiguous States into

divisions, usually consisting of groups of counties, that have similar weather conditions. As of

1987, there were 345 NOAA divisions, an average of about seven per State.

A priori reasoning suggests that method (1), using the data from the closest individual weather

station, would provide more accurate measures of degree-days. Temperatures can vary

substantially within a multi-county area, especially when influenced by changes in elevation or

proximity to large bodies of water. However, higher costs and some operating problems are

associated with method (1). Selection of the "closest" station, taking into account distance and

other relevant factors, requires manual processing operations which must be repeated for each

survey as new PSU’s or SSU’s are introduced into the sample. Data are incomplete for some

of the individual stations, so that imputation of missing data or substitution of another nearby

station may be necessary.

Based on these considerations, EIA elected to use NOAA division data on degree-days for all

survey years through 1984. Two evaluations of the effects of using alternative methods were

undertaken prior to the 1987 RECS. In a 1982 study by the Energy Resources Group (Blumstein

et al.), one site was chosen in each of the 103 PSU’s included in NIECS and its station data were

compared with averages for the NOAA division in which the site was located. The sites were

chosen to meet two requirements: high population density and presence of an individual weather

station. The data used in this evaluation were 30-year averages.

This comparison showed a median absolute difference in degree-days of five percent between the

data for the site averages and the NOAA division averages. Most of the large differences (in

excess of 13 percent) were in California, where they averaged 30 percent. Reasons for these

large differences included large divisions, with boundaries drawn to coincide with drainage basins

rather than areas of homogeneous climate, and climatic patterns that vary substantially over short

distances. The study investigators recommended that an alternative method be used to derive

degree-day values for housing units in California and in other locations that showed large

differences between individual station data and division averages.

The 1986 evaluation (Mooney and Carroll) was undertaken by the main survey contractor,

Response Analysis Corporation. It was initially limited to the five States that had shown the

largest differences between station and division data in the 1982 evaluation. Instead of selecting

one individual station to represent a PSU, a separate selection was made for each SSU. The

comparisons were based on data for the 1984 survey reference year, April 1984 through March

1985. The investigators concluded that "... using individual station data on the SSU level, rather

than NOAA divisional data, more accurately represents local temperature conditions." (Mooney

and Carroll, p.27)

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

The evaluation also examined the effect of temperature data from alternative sources on the

model-based imputation and end-use allocation procedures used in RECS. Based on the data for

the five States included in the initial evaluation, the following conclusions were reached:

... end use models run with division data are biased in several ways. First,

because of the fact that we overestimate degree days to a larger degree for low

users of the fuels, the models overestimate the amount of fuel used for space

heating. Second, because the degree to which we mis-estimate degree days varies

by household, consumption amounts at the household level are mis-estimated.

Finally, we underestimate consumption for "imputation" households because the

division model allocates too little consumption to non-space heating uses.

(Mooney and Carroll, p.40)

Subsequent to the five-State evaluation, all 1984 RECS SSU’s were assigned to individual

weather stations rather than divisions and the new degree-day values that resulted were assigned

to individual households. The models used to impute consumption and to allocate it to end uses

were rerun with the new degree-day values, and the results were compared with those derived

by using the division averages for degree-days (Response Analysis Corporation 1988). These

comparisons showed that:

• As shown in Table 5.8, there was a reduction of 3.6-percent in heating degree-days

at the national level, with particularly large reductions in the Mountain and Pacific

divisions. Conversely, there was an increase of 12.1 percent in cooling degree-days

at the U.S. level, with increases of 10 percent or more in 7 of the 9 Census Divisions.

• The changes in overall consumption were relatively small, because only households

lacking supplier data are affected.

• The only fuel with a substantial change in consumption was fuel oil, for which use of

the station data led to a reduction of 1.6 percent at the U.S. level.

• End use allocations shifted somewhat. At the national level, the use of station data

led to a 1.6 percent decline in space heating consumption which was offset by a 3.6

percent increase in water heating consumption.

As a result of the above findings, RECS degree-day data for the 1987 and subsequent surveys

have been based on records provided by NOAA for individual weather stations. The "closest"

weather station is identified for each SSU, mainly on the basis of distance, but also taking into

account differences in elevation, proximity to large bodies of water, and the extent of missing

data for the preferred station. Users should be aware that, because of this change, estimates of

degree-days from the 1987 and subsequent survey years are not directly comparable with

estimates from earlier surveys.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.8. Comparison of Heating Degree-Days Using NOAA Division Method Versus Station

Method, April 1984 Through March 1985

Census

Division

Million

Households

Heating Degree-Days Cooling Degree-Days

Division

Method

Station

Method

Percent

Difference

Division

Method

Station

Method

Percent

Difference

United States

New England

Middle Atlantic

East North Central

West North Central

South Atlantic

East South Central

West South Central

Mountain

Pacific

86.328

4.269

14.029

15.203

6.414

14.777

5.784

8.764

4.512

12.577

4,686

6,398

5,663

6,524

6,619

2,951

3,651

2,443

5,728

3,508

4,518

6,331

5,460

6,427

6,499

2,979

3,512

2,444

5,158

3,019

-3.6

-1.0

-3.6

-1.5

-1.8

0.9

-3.8

0.1

-10.0

-13.9

1,153

524

683

685

976

1,768

1,433

2,361

1,102

873

1,293

621

822

777

1,076

1,819

1,583

2,431

1,550

1,148

12.1

18.4

20.3

13.4

10.2

2.8

10.5

2.9

40.6

31.5

Source: Energy Information Administration,

Consumption and Expenditures

(May 1987).

Questionnaire and Interviewer Effects on Measurement Error

Since the inception of RECS, there have been continuing efforts to reduce measurement error by

making improvements in questionnaires and in the training and supervision of interviewers. For

each survey year, the Household Survey questionnaire and other survey instruments have been

pretested and subjected to reviews by EIA and contractor staff and other persons with expertise

in questionnaire design. As described in Section 5.1, there has been some use of in-depth

interviews in attempts to explain unusual consumption patterns. This section cites some

additional examples of relevant activities.

Pretests and Questionnaire Reviews

In preparation for the 1990 RECS, the draft Household Survey questionnaire was pretested by

three interviewers, one of them an experienced RECS interviewer, in nine households. Each of

the interviewers completed a detailed evaluation form, with comments on each section of the

questionnaire, and participated in a debriefing session. In addition to suggestions for clarifying

specific questions, some of the points raised in the overall report on the pretest (Miksovic 1989)

were (some, but not all, of these suggestions were adopted in the final version of the

questionnaire):

• The questionnaire includes some rather abrupt switches from one topic to another.

Transition sentences should be provided at these points.

• The interviewers felt that many of the questions were very wordy, especially some that

included the phrase "... other apartments, condos, households, businesses, or farm

buildings." It was recommended that the phrase "... and farm buildings" be put in

parentheses, to be used by interviewers only when it seems appropriate.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

• The format of the questionnaire made it difficult for interviewers to refer to the relevant

instructions while they were outside making and recording measurements of building

dimensions. Several possible improvements were suggested.

• Improvements in format were suggested, such as use of larger print, highlighting skip

instructions in various ways, and using different print types.

A user-needs study prior to the 1993 RECS (EIA 1993c) identified a widespread interest in the

collection of additional information on lighting, to track use of new lighting equipment

technology and to allow estimation of a measure of energy end-use intensity for lighting.

Collection of accurate information on lighting facilities and use in the home poses some

challenging cognitive problems and could require substantial additional time in the interview.

In 1992, contractor staff conducted a series of in-depth interviews, which were recorded on both

audio and video tape, to explore various means of asking for the desired data. For the first set

of interviews, seven members of the contractor staff served as respondents. Each was

interviewed twice, with a week intervening. For the second set of interviews, 5 non-staff

respondents were recruited. Two of them were a husband and wife who were interviewed

separately and then together, to attempt to reconcile differences in their reports. All respondents

were asked to report on use for the day preceding the interview. They were asked to report on

all lights (fixtures controlled by one switch, as opposed to individual bulbs) that had been used

for at least 15 minutes and to report how long each one had been turned on (Daniels 1992).

There were fairly substantial week-to-week percentage differences in reported use for the

respondents who were interviewed twice. Some of these differences could have been real; some

could have been caused by response variability. Nevertheless, the relative stability of rankings

of the seven respondents in terms of total hours of use suggested that the data could have been

used with fair reliability to classify households as high, medium, and low lighting users. The

couple who were interviewed separately and then together had significant differences, which

could not be completely reconciled in the joint interview. In general, most respondents found

it difficult to estimate the time of use for each light, and it was not obvious that allowing them

to report time of use in broad categories made it any easier than asking for an answer in hours

or fractions thereof. The room-by-room inventory approach used in the pretest was estimated

to require an average of at least eight minutes per respondent, even without additional probing

that might be necessary to obtain reasonably accurate responses.

On the basis of this test and other considerations, two sets of questions on lighting were included

in the 1993 Household Survey questionnaire. A short module, asked for all households,

requested them to report the total number of lights and the number of fluorescent lights used on

a typical November weekday for: more than 12 hours per day, between 4 and 12 hours per day,

and between 1 and 4 hours per day. A supplementary set of questions was asked only for a

subsample of 474 households. It called for a more detailed accounting covering each indoor light

used in the home, on a typical November weekday, for at least 15 minutes. Respondents were

given options on whether to report lights by room, activity, or time-of-day usage and were

allowed to report the time used for each light in actual number of hours or in class intervals

based on number of hours.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

A comparative evaluation of the quality of lighting data obtained from these two modules will

be undertaken. For the subsample of households that responded to the supplemental module, it

will be possible to analyze any differences in their responses to the two modules.

In 1992, at the request of a member of the American Statistical Association’s Energy Statistics

Committee, a survey researcher reviewed the cognitive features of the 1990 RECS Household

Survey questionnaire, with special emphasis on questions requiring respondent recall (Biemer

1992). A major finding of this review was that many of the questions in this category did not

include a reference period as a basis for the response or used a vague, unbounded or ill-defined

reference period. In response to this analysis, all recall questions planned for use in the 1993

RECS were carefully reviewed and several changes were made. For example, the 1990 question:

H-10 About how many deliveries [of LPG] does your household usually get in a year?

was changed for 1993 to:

J-14 About how many deliveries did your household get in the past 12 months?

Other Questionnaire Effects: Conservation Improvements

In the 1984 RECS, the Household Survey questionnaire included several questions about

conservation improvements that had been made to the housing unit since September 1, 1982, such

as storm doors and windows, additional insulation, caulking, weather stripping, and heating

system improvements. For all such improvements, respondents were asked to report the month

and year in which they were installed, which could have been any month between September

1982 and the month of the Household Survey interview, generally in the Fall of 1984.

Following these questions there was a general question asking whether any improvements of this

kind had been added or installed and paid for during calendar year 1983. This question was

designed to identify households eligible for the energy tax credit that was permitted on their

Federal income tax returns for that year. Households answering "yes" to this question were asked

whether or not they had actually taken the energy tax credit on their returns.

A comparison of the general question about improvements in 1983 with responses to the earlier

questions about specific improvements that were eligible for the tax credit showed that, of the

1,328 households that answered "yes to the general question, 567 (42.7 percent) had not reported

any specific improvements as being added or installed in 1983. This could have been legitimate

in some cases; the specific questions allowed for reporting of only a single month and year. If

caulking or weather stripping, for example, had been added at two different times during the

approximately 2-year reference period, only one of these would have been reported. Also, some

improvements might have been installed in 1983 but not paid for until 1984. Nevertheless, the

high incidence of apparent inconsistency suggests that responses to the questions about specific

improvements or the general question, or perhaps both, were subject to substantial bias or

response variability. The general question on improvements in 1983 was complex, with several

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

subquestions imbedded in it, and interviewers reported difficulties in administering it (EIA 1987a,

pp.112-113).

Interviewer Training

Typically, about 300 interviewers have conducted personal interviews for each of the RECS

Household Surveys. Partly because of the longer interval between surveys in recent years,

interviewer turnover between surveys has increased. In the 1987 RECS, 57 percent of the

interviews were conducted by interviewers who had not participated in the 1984 survey. In the

1990 RECS, 60 percent of the interviews were done by interviewers with no experience in the

1987 RECS. This turnover, along with numerous changes in questionnaire content, means that

the effectiveness of training and supervision of interviewers can be an important element in

determining the quality of the survey results.

For the 1980 RECS, interviewers were trained for two days in small group sessions for 10 to 20

persons each. For the 1981, 1982, and 1984 RECS, new interviewers were trained in the same

way as in 1980, but the training of experienced interviewers consisted of completion of self-

study materials and practice interviews. For the 1987 RECS, all interviewers were trained in four

large group sessions, each lasting two and one-half days.

The cost of training was becoming a substantial element in the total budget for RECS, and means

were sought to reduce training costs for the 1990 RECS. The solution adopted was to use home-

study materials for all interviewers, including a videotaped presentation in several sections, an

interviewer’s manual, a quiz and practice interviews, the last two of these to be sent in to the

survey contractor for evaluation. Use of these training methods resulted in a significant reduction

in training costs, estimated at about 30 percent in constant dollars. Part of the savings were

applied to more extensive office reviews of practice and initial interviews, followed by contacts

with all interviewers to provide feedback from these reviews (Leach 1991).

Given the rather substantial change in training procedures that was introduced in 1990, it was

considered important to try to evaluate the relative effectiveness of the old and new procedures.

Two methods were used: administration of an evaluation questionnaire to the interviewers and

comparative analyses of the extent of edit changes and imputation in the data processing.

Completed evaluation questionnaires were obtained from 257 of about 290 interviewers who

completed the training for the 1990 RECS. Most of those responding had had prior exposure to

both large and small-group training sessions for RECS or other surveys. When asked to compare

the effectiveness of and their preferences for four kinds of training--small group in-person, large

group in-person, self-study only, and self-study plus video--a large proportion, 78 percent,

thought small group in-person training was the most effective and 60 percent identified it as their

first preference. Self-study plus video was a distant second for both effectiveness (15 percent)

and preference (22 percent). When first and second ratings were pooled, self-study plus video

received favorable ratings for both effectiveness and preference from 55 percent of the

interviewers.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

The interviewers’ overwhelming preference for in-person training may be influenced in part by

factors other than its effectiveness. For most interviewers, these training sessions provide an

opportunity to travel away from their home base and to meet fellow interviewers with whom they

would otherwise not have much contact.

Interviewers were asked how well they felt they understood each of 11 topics covered in the

video presentation. Most were rated "well understood" by at least 80 percent of the interviewers.

Exceptions included:

• Fuels and equipment (61 percent understood very well). One written comment suggested

that the treatment of combination equipment was inadequate.

• Measurement of total and heated floorspace. This has always been one of the most

difficult aspects of the survey for interviewers. Some of the written comments praised

this section of the videotape, indicating it was more realistic than what could have been

done in a classroom training session.

• Recording of air-conditioner nameplate data. This was a new requirement for the 1990

RECS, so it was difficult to anticipate what kinds of problems might arise and discuss

them in the training.

In response to a question about the degree of difficulty of the training exercises and final quiz,

nearly all of the interviewers considered them "about right." However, many interviewers did

quite badly on the exercises and quiz, especially on topics such as what to measure and what not

to measure, who are eligible respondents, and households versus group quarters. These same

topics had caused many problems in training and in actual field work in prior survey years.

In addition to finding out how interviewers reacted to the new training procedures and what

improvements they had to suggest, it was felt important to seek an objective measure of the

effects of the new procedures on actual interviewer performance. The method of analysis

adopted was to compare the levels of edit and imputation changes for 14 RECS variables for the

1987 and 1990 RECS in total and for experienced and inexperienced interviewers in each survey

year. An experienced interviewer was defined as one who had participated in RECS in the

immediately preceding survey year. Inclusion of interviewer identifiers on the RECS file of

individual household data for each survey year made it possible to distinguish the work of

experienced and inexperienced interviewers. The 14 variables were chosen from among those

that were included in the same form in both survey years, had extensive edit checks, and had

required the most imputation (Response Analysis Corporation 1991).

The indicator of interviewer performance used in the analysis, admittedly an indirect measure,

was the proportion of sample households for which changes had been made in each of the 14

variables following data entry, as the result of editing and imputation procedures. Changes were

detected by comparing initial and final values for each variable; intermediate changes were not

taken into account. Interviewer errors were only one source of such changes; they could also

result from respondent and data entry errors. Moreover, some interviewer errors were detected

and corrected in manual reviews prior to data entry.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.9 shows, for each of the 14 variables selected, the percent of sample households with

changes for each survey year for experienced and inexperienced interviewers. Overall, the data

do not reveal a substantial difference between the change indicators for the 1987 and 1990

surveys. For a few variables there were fairly large differences in the proportion of changes

between the two years: same heating fuel as prior survey (down 5 percent in 1990), year house

built (down 6 percent), and presence of a basement (up 4 percent). Except for the possibility of

insufficient attention in the 1990 training to the item on basements, more detailed analyses did

not reveal any obvious reasons for these changes.

As shown in Table 5.9, the mean proportion of changes for the 14 variables was slightly higher

for inexperienced interviewers in both survey years. This finding does not necessarily prove that

their performance was not as good as that of experienced interviewers. An alternative

explanation would be that interviewer turnover is larger in areas like central cities, where there

is likely to be a higher incidence of respondent error and item nonresponse requiring imputation.

Table 5.10 provides data relevant to this hypothesis. As shown in the last column of that table,

there is a clear association between housing characteristics and the extent of changes subsequent

to data entry. Households in center cities, those living in apartment buildings, and those who

were not owners had the largest number of changes. These explanatory variables are correlated,

and some of the differences for renters of apartments may be explained by changes made to

variables related to heating fuel and equipment based on responses to the Rental Agent Survey.

However, the data in Table 5.10 on percent of interviews completed by experienced interviewers,

by housing type, provide only moderate support to the supposition that there are higher

proportions of inexperienced interviewers conducting interviews with the types of households for

which changes subsequent to data entry are most frequent. The proportions of experienced

interviewers are nearly the same for owner and non-owner occupied housing units. They were

slightly lower for apartments than for single-family units and lower for households living in

metropolitan areas.

Taken overall, the results of the interviewer questionnaire and the analysis of processing changes

did not demonstrate any clear or substantial differences in effectiveness between the 1987 and

1990 RECS training. However, two features of the 1993 RECS seemed to favor the use of in-

person over self-study training for that survey: first, the inclusion of several new items in the

Household Survey questionnaire and, second, as a result of the 1993 sample revision, 22 out of

116 primary sampling units had not previously been included in RECS, so the proportion of

interviewers lacking previous RECS experience was likely to be higher than usual. Consequently,

two and one-half days in-person training sessions were held for all interviewers at four sites,

followed by a small make-up session and some telephone training for replacement interviewers.

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.9. Percent of Household Survey Interviews with Imputation or Edit Changes Subsequent

to Data Entry, by Interviewer Experience

: 1987 vs. 1990

RECS 1990

Question

Total 1987 RECS 1990 RECS

1987 1990 Exp. Not Exp. Exp. Not Exp.

B-2 Main Heating Fuel

B-9 Main Heating Equipment

C-3 Water Heating Fuel

K-10 Income

B-3 Same Heating Fuel as 1987

I-1 Budget Plan for Main Fuel

A-6 Year House Built

P-11 Has Basement

L-12 Public Housing

K-7 Race

K-1 Non-Householder Age

P-12 Amount of Basement Heated

K-1 Householder Age

K-6 Marital Status

Mean: 14 Items

4.3

4.5

4.0

4.4

4.3

4.6

Number of Interviews 5,856 4,828 2,530 3,326 1,965 2,873

* = Rounds to less than 0.5 percent.

Interviewers were counted as experienced if they had worked on the immediately preceding RECS.

The 1987 question was "Same Heating Fuel as 1984."

Note: Exp. = Experienced Not Exp. = Not Experienced

Source: Response Analysis Corporation,

Quality Assessment of Videotape Training: Conclusions and Recommendations

(September 1991).

Energy Information Administration / Energy Consumption Series

Residential Energy Consumption Survey Quality Profile

Table 5.10. Interviewer Experience and Extent of Edit and Imputation Changes, by Type of

Household: 1990 RECS

Household