PG&E`s Emerging Technologies Program
Contents
1. Other notable comments included The app sometimes had trouble connecting to the thermostat Some customers would like an explanation of the app s features in the user manual Several customers noted that programming the thermostat was confusing compared with the same process on the website Some customers were annoyed about having to re enter their password after updates and unexpected log outs and Many customers would like to see their savings and or their bill displayed on the iPhone app VWeBSIIE PORTAL Of the customers who commented 98 gave the website portal a rating greater than 5 In general there was a split in responses About half of the comments showed disinterest or a lack of opinion It appears many customers didn t use the website because they preferred the iPhone app or they didn t know about it The other half of the comments gave positive responses noting the ease of use and convenience of remote access 19 PG amp E s Emerging Technologies Program REC BEVING MESSAGES FROM WEBSITE AND IPHONE APP POSITIVE OR NEGATIVE EXPERIENCE Approximately 80 of the comments were from customers who rated the experience 5 and above Overall 65 of survey participants responded to this question 45 of the responders liked the messages and found them to be useful reminders 26 of responders didn t receive any messages 22 had no opinion and 13 disliked them Note that those designations are not mu2. PG amp E s Emerging Technologies Program TABLE 7 RANKINGS OF POSSIBLE ENERGY REDUCTION FACTORS Ranki Least A anking eas Setting up my temperature profile upon enrollment 16 35 16 1 Remote access to my thermostat through my iPhone 48 19 1 0 The messages have received 3 11 46 8 Other 1 3 5 59 If no Other criteria was specified it was automatically set to 4 Several customers specified their Other category and the most popular answers were e Being able to compare usage to others usage e Ease of use and increased active participation and e Increased awareness of energy usage Customers were then asked to answer the following question How could PG amp E make it easier for you to track your energy usage More than half of the survey participants did not respond but among those that did the responses were similar Customers want to be able to track their usage savings alternative energy production what tiers they are in and be able to see this alongside the comparison data already provided in the app as well as their own historical usage data Some would like it daily on their phones or thermostats Others would rather have a monthly report delivered via email ISSUES Customers were asked to answer the following question Have you had any problems connecting with or controlling your thermostat remotely Fifty out of 145 respondents answered yes in which case they were asked to describe
3. customers from enrolling in the program Enrollment questions have been revised and clarified and the installation rate is expected to increase 29 PG amp E s Emerging Technologies Program APPENDIX A SURVEY INSTRUMENT Smart Thermostat Survey Introduction Thank you for taking the time to participate in this program and completing this survey This survey should take less than 10 minutes to complete We encourage your candid feedback and there are many OPTIONAL open ended questions to enable you to expand on your impressions Your responses will remain anonymous and will be used for research purposes only To begin please click on the button below 1 Did you feel that your enrollment process for the pilot was a positive experience Using a 10 point scale would you say you had a positive customer experience 10 or a negative customer experience 1 1 2 3 4 5 6 7 8 9 10 Negative customer Positive customer experience experience 2 Why did you give a rating of insert rating from above for the enrollment process Please be as specific and detailed as possible open end and not required 3 How did you enroll to participate in the Smart Thermostat Pilot a With the help of a staffer at one of the booths b On my own at home through the online enrollment website c Other 4 Please tell us how important each of the following reasons was to you in deciding to enroll in this program
4. scale will be displayed for each reason with options Very Important Somewhat Important Somewhat Unimportant and Completely Unimportant Attractiveness of the thermostat Usefulness of the information provided by the thermostat The potential for energy savings Ability to remotely monitor and control the thermostat My existing thermostat was broken My existing thermostat was obsolete The product is new and different Saving money Other Specify mre ho Ao oP 5 We are interested in your overall impression of the thermostat to date Using a 10 point scale would you say you like this thermostat very much 10 or do not like this thermostat 1 1 2 3 4 5 6 7 8 9 10 Do not like this Like this thermostat thermostat very much a 6 About how often have you tried using the iPhone app to control your thermostat a More than 16 times b 6 15 times c 2 5 times d Once 30 PG amp E s Emerging Technologies Program e never 7 If not e We are interested in your impression of the iPhone app Again using a 10 point scale would you say you like the iPhone app very much 10 or do not like the iPhone app 1 1 2 3 4 5 6 7 8 9 10 Do not like the iPhone Like the iPhone app app very much 8 if prior question answered Why did you give a rating of insert rating from above for the iPhone app Please be as specific and detailed as possible o
5. Program TABLE 8 IF YES HOW WOULD YOU LIKE TO RECEIVE ADDITIONAL INFORMATION CHOOSE ALL THAT APPLY Method No By mail in the same envelope 3 37 as my PG amp E Bill By mail but in a separate 3 37 envelope from my PG amp E bill Via email 32 8 Online at pge com 3 37 Mobile application on your 18 22 iPhone Other Specify 1 39 The most popular option was to receive additional information via email The second most popular method was through the iPhone app Most customers did not prefer the mail and online option The comments listed under the open ended response showed a desire for some kind of manual for the app However many customers asked specific questions pertaining to their individual issues so perhaps it would be more beneficial to implement a bulletin or forum style help resource where specific and commonly asked questions could be answered Many customers also expressed a desire for examples and tips Overall it appears that customers wanted an introduction that included examples and tips as well as some source of feedback for specific issues Several customers wanted updates on new features and information about the future of the program Finally participants were asked whether they would be willing to be contacted by PG amp E to share their experience during the trial Sixty four of 145 respondents answered yes to this question PARTICIPANT C HARAC TERISTICS Although the full test populatio
6. RCT design to produce accurate and precise estimates of savings over a one year period Also of interest are several aspects of the customer experience which include e How Satisfied participants were with the recruitment and installation process e How much they liked the thermostat device and how useful they found it e How much they liked the iPhone and PC apps and how useful participants found the apps e Whether participants liked the messages that were sent to them through their iPhones and e Whether participants thought that the technology would lead them to save energy Relevant data for these issues has been gathered using surveys and has been evaluated mainly qualitatively to assess the technology and its installation process FIELD TESTING OF TECHNOLOGY This is a test of a thermostat technology its associated iPhone and PC apps and the installation process for all three The sites for testing are homes of PG amp E residential customers that satisfy particular eligibility criteria that would be similar to the criteria a customer would have to satisfy to make use of the thermostat if it were offered through retail Therefore this test bed provides unbiased results Test sites may have a variety of different incumbent technologies ranging from traditional thermostats to other types of programmable thermostat The system is controlled by the participant TEST PLAN There is no baseline measurement for comparison The
7. Treatment Group lt Active Thermostats 1000 900 WE 600 os N 500 One other source of information on the characteristics of installed customers is a short demographic survey that Honeywell s technicians had customers fill out during installation It contained six questions covering the number of stories in the customer s home the number of bedrooms the number of adults who live there the number of children who live there the age of the head of household and the education level of the head of household About 60 of the homes have 1 story and 36 have 2 stories Over 80 of the homes have 3 or 4 bedrooms About 60 of homes have 2 adults with about 15 each having 1 or 3 About 45 of the homes have no children about 16 have 1 and 30 have 2 About 35 of heads of household are 35 44 years old with 23 in the range of 25 34 and 20 in the range of 45 54 The most common education level of heads of household is bachelor s degree with 36 giving that response Next is a master s degree with 25 followed by associate s degree and some college with no degree at 11 and 13 respectively Finally Table 10 shows a comparison of treatment and control customers along with estimated p values of tests for statistically significant differences A p value is a metric that indicates how likely it would be to see differences between groups of the indicated size if the groups were actually drawn from the same und
8. had used the website that often More than half of respondents had never used the website portal Just over half of the respondents reported either never changing their thermostat settings due to the messages they received or never having received any messages Customers were also asked to answer the following question Do you believe that you have reduced your energy use as a result of the thermostat and messaging provided through this program Eighty nine customers responded Yes or No Don t Know was also an option Of these 69 78 said Yes and 20 said No If customers answered no they were asked to leave comments as to why The most common answers were Pre existing energy conservation habits or a previous thermostat already helped them maintain efficient energy levels They have not used the thermostat app very frequently and The new device resulted in greater energy use due to difficulties programming or software malfunctions If customers answered yes regarding their belief that they have reduced energy use they were asked to rank the possible reasons why they think they were able to reduce usage and were also asked to leave comments regarding their rankings Table 7 summarizes the results The ranks went from 1 to 4 with 1 meaning Most Important and 4 meaning Least Important The factors ranked most important were remote access to the thermostat followed by setting up the temperature profile at enrollment 22
9. implicit technology baseline is each participant s previous thermostat and the test will rely on participants to make that comparison The energy consumption savings associated with the HVAC being controlled by the test technology is not measured in this report but will be estimated in a future report The variables used to measure performance are responses to survey questions about the technology and its associated installation process These questions are detailed in the results section The monitoring period for the trial thus far is approximately three months Data are gathered via online and paper surveys 12 PG amp E s Emerging Technologies Program RESULTS This section is divided into three subsections describing results from customer recruitment survey findings and characteristics of participating customers thus far A copy of the survey instrument is provided in Appendix A CUSTOMER REC RUITMMENT AND THERMOSITATINSTALLATION Customer recruitment was performed by GoGorilla Media a subcontractor to Honeywell The recruitment mechanism began with a series of 250 retail intercept events which took place outside retail stores malls festivals and farmers markets in Contra Costa San Joaquin Yolo and Solano counties These events were staffed by five two person teams and took place over the period J uly 19 to September 30 2012 At these events representatives of GoGorilla would stand at a promotional display for the th
10. than 8 to respond Out of that group only nine customers gave comprehensible answers The majority of responses were related to not getting questions answered and included Representatives did not get back to them about their inquiries Representatives did not understand their problem or couldn t answer their questions and Representatives provided unclear explanations to questions Other responses had to do with defective equipment and slow responses in fixing equipment malfunctions DEG REE OF AGREEMENT QUESTIONS The survey gave participants a list of questions and asked them whether they agreed or disagreed with them Table 5 summarizes the responses to those questions It shows the tally of the number of times respondents gave each rating on a scale from 1 to 10 with 1 meaning Strongly Disagree and 10 meaning Strongly Agree 20 PG amp E s Emerging Technologies Program TABLE 5 TALLIES OF RESPONSES TO DEGREE OF AGREEMENT QUESTIONS Strongly Strongly Disagree i 3 Question 8 Rating 1 10 Agree Average Rating The thermostat iPhone app and website alolalalo portal have met my expectations The thermostat iPhone app and website portal give me greater control over my 2 2 2 1 6 5 8 15 30 74 8 7 energy usage would recommend the thermostat and 3 o 3 21 8141 7 101 34 1 74 8 7 app to a friend The thermostat on the wall is easy to use 3 3 4 6 9 8 16 17 29 50 7 9 The iPhone ap
11. their problems The most commonly occurring problem had to do with connectivity issues or trouble recognizing the wireless network The second most common problem was issues with the app itself For example sometimes the app would not register the temperature change or would say it had executed the change when obviously it had not The app sometimes crashed would not open or wouldn t let them make changes etc Also noteworthy is that quite a few of these issues were resolved on their own Customers were then asked Did you need to contact customer service for support at any point This question was not limited to those who had answered yes to the previous question Twenty one of 145 respondents answered Yes in which case they were asked to describe their problems The majority of people called about connectivity issues Several customers called to ask about the operation of the thermostat or app ADDITIONAL SUPPORT MATERIALS Customers were asked if they would like to receive additional educational communications or information regarding their thermostat the iPhone app or the website portal Forty of 145 answered yes in which case they were asked what kind of information or communication they would like to receive and how they would like to receive it This question was asked both as an open ended question and with multiple choices The responses to the multiple choice question are summarized in Table 8 PG amp E s Emerging Technologies
12. 20 Frequency of Use and Effect 22 Issues 23 Additional Support Materials 23 Participant Characteristics 24 EVALUATIONS 29 RECOMMENDATIONS 29 APPENDIX A SURVEY INSTRUMENT 30 Pacific Gas and Electric Company PG amp E s Emerging Technologies Program EXEC UTIVE SUMMARY This emerging technology report documents the results from the initial installations of the Honeywell communicating thermostat and Opower behavioral software trial The thermostat includes programming that allows it to be remotely programmed through an iPhone app or a PC based app It also has the functionality to push messages to participants through the iPhone app alerting them to important aspects of their energy use In this trial 888 customers have been recruited into the trial program through a series of marketing events conducted outside selected retail stores malls festivals and farmers markets in Contra Costa Yolo San Joaquin and Solano counties Of those 888 customers 824 customers completed online enrollment and were randomly assigned to treatment and control groups 421 were randomly assigned to be in the treatment group and made appointments for Honeywell technicians to install the trial thermostat at their home Of those 276 had successful thermostat installations The remaining 403 were randomly assigned to be in a control group for evaluation measurement and verification EM amp V purposes This control group has not and will not rec
13. 868 0 27 Aug 11 886 838 0 14 Sep 11 865 826 0 19 Oct 11 710 683 0 26 Nov 11 759 747 0 63 Dec 11 Average kWh 860 826 0 26 Jan 12 818 796 0 43 Feb 12 717 695 0 32 Mar 12 672 661 0 59 Apr 12 698 670 0 23 May 12 750 702 0 07 Jun 12 886 838 0 15 It is also worth noting one other fact about the table although every sign indicates that the experiment is progressing well there are still noticeable differences in average usage between the treatment and control groups that fluctuate between 2 and 6 of monthly PG amp E s Emerging Technologies Program usage That is the difference between the groups changes by 4 percentage points due to chance alone during the year before the program This underlines the need for large samples to detect small effect sizes and the need to increase the size of this test before performing impact estimates If the expected energy savings from this thermostat is on the order of 5 then it would be hard to have confidence in estimated impacts based on the experiment at its current size This is particularly true given that it is not expected to see a full 5 effect in the treatment group due to treatment group premises where installation does not take place When the underlying differences between groups fluctuate with the same magnitude as the effects being measured then the risk is higher that estimated impacts are significantly affected by random variation in the data EVALUATIONS The main results of the eva
14. Factors 23 Table 8 If Yes How would you like to receive additional information Choose all that Apply eeeeeeee 24 Table 9 Customer Characteristics cceceeeee eee e eee ee ee eee ee ee eee ee 26 Table 10 Treatment Control Comparison cceeeee scene ee eee ee ee 28 FIG URES Figure 1 Treatment Control and Installed Customers 0000005 14 Figure 2 Locations of Recruiting and test groups eeeeeee ee 25 Figure 3 Customer Monthly UsSage cceeeee eens eset ee ee eee eeeeeed 27 Pacific Gas and Electric Company PG amp E s Emerging Technologies Program ET11PGE3073 CONTENTS EXEC UTIVE SUMMARY 5 INTRODUCTION 8 BACKGROUND 8 EMERGING TEC HNOLOGY PRODUCT 10 ASSESSMENT OB ECTIVES 10 TECHNOLOGY PRODUCT EVALUATION 11 TECHNICAL APPROAC H TESTMETHODOLOGY 12 Field Testing of Technology 12 Test Plan 12 RESULTS 13 Customer Recruitment and thermostat installation 13 Survey of Participants 15 Motivation for Participation and Method of Enrollment 16 Motivation for Participation 16 Method of Enrollment 17 Overall Impression and Helpfulness Responses and Comments 17 Enrollment Process 19 iPhone App 19 Website Portal 19 Receiving Messages from Website and iPhone App Positive or Negative Experience 20 Messages from Website and iPhone Helpful or Unhelpful 20 Customer Service 20 Degree of Agreement Questions
15. How would you like to receive the additional information Choose all that apply Randomize options d By mail in the same envelope as my PG amp E bill By mail but in a separate envelope from my PG amp E bill Via email Online at pge com Mobile application on your iPhone Other Specify pro 32 Do you believe that you have reduced your energy use as a result of the thermostat and messaging provided through this program Yes no 33 If no please provide feedback on why you feel the thermostat and messaging have not helped you reduce your energy use open end and not required 34 If yes please rank in order of importance the following reasons why you think you have been able to reduce your energy use where 1 is the most important reason and 4 is the least important reason Reason Ranking Setting up my temperature profile upon enrollment Remote access to my thermostat through my IPhone The messages I have received Other Please explain 35 If you have any comments about the ratings you just provided please note them here Please be as detailed as possible open end and not required 36 How could PG amp E make it easier for you to track your energy usage 37 What other feedback if any do you have concerning your participation in this program Optional Open end and not required 33 PG amp E s Emerging Technologies Program 38 Optional If you are willing to be contacted by PG amp E regarding media opportuniti
16. PG amp E s Emerging Technologies Program ET11PGE3073 Opower Honeywell Thermostat Trial Interim Findings ET Project Number ET11PGE3073 bade Meet your new Thermostat Manage your temperature settings with the FREE iPhone app and convenient remote control thermostat Now that s smart Product Manager Olya Zaretsky Pacific Gas and Electric Company Prepared By Michael Perry and Jeeheh Oh Freeman Sullivan amp Co 101 Montgomery St 15 floor San Francisco CA 94104 Issued December 12 2012 Copyright 2012 Pacific Gas and Electric Company All rights reserved Pacific Gas and Electric Company PG amp E s Emerging Technologies Program ET11PGE3073 ACKNOWLEDGEMENTS Pacific Gas and Electric Company s Emerging Technologies Program is responsible for this project It was developed under internal project number ET11PGE3073 Michael Perry of Freeman Sullivan amp Co conducted this technology evaluation for Pacific Gas and Electric Company with assistance from Jeeheh Oh and overall guidance and management from Olya Zaretsky For more information on this project contact OxZ1 pge com LEGAL NOTICE This report was prepared for Pacific Gas and Electric Company for use by its employees and agents Neither Pacific Gas and Electric Company nor any of its employees and agents 1 makes any written or oral warranty expressed or implied including but not limited to those concerning merchantabil
17. at due to the experimental design of the trial not every customer will receive a thermostat Figure 1 shows the trajectory of recruitment and installation from the beginning of installations in early August through October Beginning in early August the treatment group control group and installed group each increase steadily until late September when the retail intercept events were paused to assess how future marketing should be done 13 PG amp E s Emerging Technologies Program FIGURE 1 TREATMENT CONTROL AND INSTALLED CUSTOMERS Treatment Group Control Group t Installed Group 450 400 350 300 N uw oO Customers N O O 150 100 oo o O oooO O A O AAE C ae ora a lt 2 50 0 7 30 2012 8 9 2012 8 19 2012 8 29 2012 9 8 2012 9 18 2012 9 28 2012 10 8 2012 10 18 2012 10 28 2012 Date Installation of the thermostat was performed by Honeywell installers who also asked customers to fill in a satisfaction survey specifically about the installation experience In the survey customers rated several aspects of the installation on a scale from 1 10 with 10 being the highest rating The average responses from this survey are shown in Table 1 below Overall customers gave very high ratings with only a single average value below 9 at 8 9 TABLE 1 AVERAGE RESPONSES TO INSTALLATION SURVEY Overall field service rep satisfaction Likelihood to recommend 9 1 On
18. eive the thermostat They are included in the project in order to provide a benchmark set of usage data for customers with characteristics comparable to those who were assigned to receive a thermostat They have comparable characteristics because they were chosen randomly from the group of customers who were eligible for and interested in receiving the thermostat Throughout this report the term treatment group refers to all customers who were randomly assigned to receive a thermostat whether installation was successful or not The term installed group refers to the group within the treatment group for which installation was successful The term control group refers to the customers who were determined to be eligible for the trial and wanted the thermostat but who were randomly assigned not to receive a thermostat Honeywell technicians installed the devices and made sure that participants iPhones were set up with the app Participants received instructions on how to use the device and the apps Recruiting has not yet finished so this report only documents results for a portion of what will ultimately be the full test population With that in mind the treatment and control populations that have been recruited and assigned thus far show no signs of selection bias which is an indicator that the energy savings estimates that ultimately come from this trial will be accurate Customers with successful installations were surveyed over the interne
19. erlying distribution In a table such as this we should be concerned about selection bias if any p value is around 0 02 or less or if several are 0 1 or less Neither of those conditions is true here This table is a standard 7 P values for comparisons of means are based on t tests P values for categorical comparisons baseline zones and income categories are based on chi squared tests 27 PG amp E s Emerging Technologies Program method for demonstrating the validity of an RCT The fact that the groups are highly comparable and that p values in the table are quite high is a good sign that the randomization has worked as expected thus far and the experiment is valid TABLE 10 TREATMENT CONTROL COMPARISON Category Unit Control Treatment P Value Zone P 0 0 ZoneQ 0 0 Zone R 0 0 Zone S 37 32 Zone T 0 0 0 37 Zone V 0 0 Zone W 0 0 Zone X 63 67 Zone Y of Group 0 0 Zone Z 0 0 lt 30k 17 15 30k S50k 6 5 me 50k 80k 19 20 gt 80k 58 61 CARE 11 10 0 79 Rent 1 0 0 15 Pool 29 30 0 68 Residents Average number of residents 3 0 3 2 0 19 Living Space Average square feet 1983 2020 0 48 Year Built Average year built 1976 1975 0 54 Adult Age Average head of household age 50 50 0 65 Jun 11 875 831 0 21 Jul 11 904
20. ermostat and invite people to talk to them about it They would then attempt to persuade customers to sign up for the trial Customers who agreed to sign up entered their personal information into an onsite computer including their PG amp E account information After a set of qualifying questions on the computer to determine that the customer was eligible and willing to take part and questions about their individual home thermostat settings the customer was randomly assigned to either the treatment or control group Customers in the treatment group then made an appointment to have a Honeywell technician come to their home to install the device Trial eligibility requirements are fairly restrictive particularly the requirement to own an iPhone and presented a challenge to recruiting Survey results discussed later in this report showed a high degree of satisfaction with the recruitment process However the survey was conducted only among customers who had a thermostat successfully installed which may pre dispose those customers to approve of the recruitment process in retrospect There is some anecdotal evidence of customer dissatisfaction with spending time to sign up for the trial only to be put into the control group FSC is currently not aware of any efforts to quantify how often customers became upset when that occurred There is currently a greater effort by GoGorilla to better educate customers about the trial and set expectations th
21. es all those customers who were originally assigned to the treatment group to be retained in the treatment group regardless of whether they receive a thermostat This will lead to greater variance in the treatment group without providing additional measurement of energy use in the presence of the thermostat Thus these missed installation opportunities lead to a less accurate energy savings estimate for any given treatment group size For the second phase of recruitment efforts have been made to increase customer education about eligibility requirements and trial details in order to avoid instances in which an installation was unable to be completed for a treatment group customer Additional information about the information process has been provided by Honeywell technicians Notable aspects of the installation process include e Using a repeater is not very helpful for fixing thermostat wireless connectivity issues even when more than one was added e Of customers with a pre existing programmable thermostat approximately 50 did not have it programmed e Technicians spend an average of 20 30 minutes with customers to explain the leave behind materials and demonstrate the functionality of the app e The most common questions technicians encounter from customers is how do change my setting from heat to cool and e 90 of customers prefer to be shown how to engage their thermostat with their smart phone only as opposed to being sho
22. es to share your experience during the Smart Thermostat Pilot please check the box below It does not guarantee you will be contacted and you have the right to decline to participate if contacted Such opportunities may involve interviews photographs etc Name Phone Number That is the end of our survey Thank you very much 34
23. ges 10 or did not like receiving the messages 1 1 2 3 4 5 6 7 8 9 10 Did not like receiving the Liked receiving the messages messages 15 if prior question answered Why did you give a rating of insert rating from above for whether you liked the messages Please be as specific and detailed as possible open end and not required 16 How often did you proactively change your thermostat settings as a result of receiving messages on your iPhone app or website portal More than 16 times 6 15 times 2 5 times Once Never I did not receive any messages mo aogs 31 PG amp E s Emerging Technologies Program 17 Please indicate whether you agree or disagree with the following statements using a 10 point scale where 1 means you strongly disagree 10 means you strongly agree and you can use any number between and 10 Randomize statements 1 2 3 4 5 6 7 8 9 10 Strongly Strongly disagree agree The thermostat iPhone app and website portal have met my expectations The thermostat iPhone app and website portal give me greater control over my energy usage would recommend the thermostat and app to a friend The thermostat on the wall is easy to use The iPhone app is easy to use The website portal is easy to use The iPhone app and web portal provide useful information I m glad I signed up for the program
24. haracteristics of customers recruited and the customer experience after 2 3 months of using the device This report documents and summarizes that data The remainder of the report contains background on the trial followed by a description of the product being tested and then a discussion of how data has been collected followed by a discussion of results so far The report concludes with a summary and recommendations BACKGROUND Honeywell and Opower Inc have collaborated to develop a solution that includes an internet enabled thermostat and behavioral software that allows residential customers to better manage their climate control energy use The thermostat and software make it easier for a customer to allow the temperature in their home to change to more efficient levels when they aren t home or when they are sleeping When setting their at home and during sleep cooling and heating temperatures customers are given a prompt if any setting is above or below the recommended setting They are prompted to set the temperature higher or lower because the home will have cooled or warmed up to a comfortable setting before they wake up They are also informed of how much money they would save either during winter or summer if they decided upon the recommended temperature setting Similar prompts appear when setting the temperature for when they are normally away from home Customers may also use a manual override to make changes to their normal schedu
25. ingle family home townhouse or condo e Have only one zone for climate control one thermostat in the home and e Have high speed internet at their home with an available port on their router Additionally in order to be assigned to receive a thermostat they had to be randomly assigned to the treatment group after eligibility was determined The assessment of the trial so far has been performed by a team including staff from both PG amp E and FSC FSC is a contractor providing advice and assistance with many aspects of the trial including sampling recruitment and installation FSC has extensive experience in designing and evaluating technology and behavior based energy efficiency trials FSC will be the evaluator that estimates energy savings later in the trial Of relevance to much of this report is that FSC also has extensive experience in survey research and focus groups evaluating customer facing programs in the utility industry 11 PG amp E s Emerging Technologies Program TECHNICAL APPROACH TEST METHODOLOGY The technical approach to the test is that of a randomized controlled trial RCT in which the primary data of interest are the energy savings induced by the thermostat in the treatment group as compared to the control group This will be evaluated in a future report after more customers have been recruited and after more time has passed for energy savings to be observed The evaluation of this savings will make use of the
26. ity or fitness for a particular purpose 2 assumes any legal liability or responsibility for the accuracy completeness or usefulness of any information apparatus product process method or policy contained herein or 3 represents that its use would not infringe any privately owned rights including but not limited to patents trademarks or copyrights Pacific Gas and Electric Company PG amp E s Emerging Technologies Program ET11PGE3073 ABBREVIATIONS AND ACRONYMS HVAC Heating Ventilation and Air Conditioning PC Personal Computer App Software application either on an iPhone or a PC RCT Randomized controlled trial Pacific Gas and PG amp E s Emerging Technologies Program ET11PGE3073 TABLES Table 1 Average Responses to Installation Survey c sceseeees 14 Table 2 Frequency of Responses to Motivation for Participation Questions and Average R SPONSE 6 eeceeeeeee ee eeeee es 17 Table 3 Frequency of Responses to Overall Impression Questions and Average R SPONsSe ceceee cence eee eset ee etna ea eeees 18 Table 4 Frequency of Responses to Helpfulness Questions and Average RESPONSE cece eee nett eee eee eee eeneeeneeeneeenee ene 18 Table 5 Tallies of Responses to Degree of Agreement Questions 21 Table 6 Frequency of Responses to Frequency of Use and Effect QUESTIONS eirias ra ninaa E EEEREN 22 Table 7 Rankings of Possible Energy Reduction
27. l Open end and not required 25 During installation you should have been provided with a manual for the thermostat and a one page introduction to the iPhone app and website portal Do you recall receiving those documents a Yes b No if no then skip to Q32 32 PG amp E s Emerging Technologies Program 26 Did you look at those documents a Yes b No if no then skip to Q32 27 if yes to Q25 We are interested in your impressions of each of these First did you feel that the thermostat manual was helpful Using a 10 point scale would you say you found it very helpful 10 or not at all helpful 1 1 2 3 4 5 6 7 8 9 10 Not at all helpful Very Helpful 28 Next did you feel that the one page introduction to the iPhone app and website portal was helpful Using a 10 point scale would you say you found it very helpful 10 or not at all helpful 1 1 2 3 4 5 6 7 8 9 10 Not at all helpful Very Helpful 29 Besides the support materials left at your home after installation would you like to receive additional educational communications or information regarding your thermostat the iPhone app or the website portal a Yes b No c Don t know 30 If 29 is yes What additional communications or information would you like to receive Please be as specific and detailed as possible open end and not required 31 If 29 is yes
28. le Ifa customer chooses a more efficient temperature compared to their present schedule they are asked if they would like to adopt the new temperature as their present temperature For example the message may read Would you like to save 79 as your usual temperature for Home Not saving this temperature could cost you an extra 20 this year A Opower s original proposal document for the trial Pacific Gas amp Electric Honeywell Opower Thermostat Pilot July 2011 prepared by Sean Harrington contains an analysis estimating total annual electricity savings of 573 kWh for premises in the Sacramento area This is roughly 5 8 of annual average usage for customers in hotter areas of PG amp E s territory The same analysis also estimates 40 therms of gas savings per year which is roughly 8 9 of annual gas usage in the same area PG amp E s Emerging Technologies Program Additionally the software provides messages pushed to customers through the smart phone app Push notifications can be sent five minutes before the heat or cooling system turns on to begin the switch from Away On Vacation to At Home temperature settings This notification allows users to set a later time for the system to activate if they re not going to be home soon Users can also receive three different push notifications regarding their fan usage a 2 hour 8 hour and 24 hour usage warning The message informs users that their fan has been r
29. llation was attempted in the customer s home In these cases ineligibility was due to reasons such as incompatible HVAC the HVAC system not functioning an ineligible network or phone In order to further preclude ineligible customers from enrolling in the program enrollment questions have been revised and clarified This action should greatly increase the volume of the installed group PG amp E s Emerging Technologies Program INTRODUCTION This report describes the procedures and results obtained in the first installation phase of the Opower Honeywell solution trial The results from this evaluation will support PG amp E Honeywell and Opower in the further execution of this trial There is much interest in the utility industry in using technology and information treatments to induce behavior change that leads to energy savings A particular area of interest lately is in using software enabled thermostats to help customers make more efficient climate control decisions The thermostat solution tested as part of this trial is an example of this More information about the thermostat is in the Background section below It is expected that the thermostat may be able to induce a 5 electricity and somewhat larger gas savings among customers who live in areas where climate control represents a large portion of their energy use The trial thus far has produced data that allows us to understand the recruitment and installation process the c
30. lled Control i Recruitment Counties Table 9 shows a comparison of the PG amp E residential population receiving electricity service to the overall test group and to the group that has successful thermostat installations The characteristics shown in the table are the fraction of each group in each of PG amp E s baseline zones which determine bill levels and which are highly correlated with climate the fraction that are in each of four income brackets as determined by data gathered by Targetbase the fraction that are on the CARE low income rate the fraction that rent their home the o Targetbase is a database marketing research firm based in Dallas Texas 25 PG amp E s Emerging Technologies Program fraction that have a pool the average number of residents the average living space in the home the average year the home was built and the average age of the head of the household The last six variables from renting on were also determined by Targetbase As expected customers in the test group come from only two baseline zones which was part of the trial s design On average test group customers have significantly higher income than the PG amp E residential population which is expected given the eligibility criteria for the trial They own their home are much more likely to have a pool and typically have more living space than the broader PG amp E population Differences in characteristics between the test grou
31. luation are that e The experimental aspect of the program is progressing well although more customers are needed for an accurate energy savings estimate e Customers have generally had a positive experience with the device thus far and gave quite positive responses to whether they agreed that they were glad they had signed up for the trial and whether they would recommend the thermostat to a friend e Customers very much appreciate the ability to control their thermostat remotely and highly prefer the iPhone app to the website for doing so and e Customers in the trial are quite different from the broader PG amp E population in fairly predictable ways given the eligibility criteria RECOMMENDATIONS Recommendations from the trial thus far are to continue to recruit until the test group reaches a size appropriate for estimating energy savings This will entail a minimum of 500 installed thermostats Recruitment would be easier if an app were available for smart phones rather than only iPhones and if customers were offered a consolation prize if they were randomly assigned to the control group Every effort should be made to maximize installation success at treatment group premises 8 An Android app has been released for customer use and is an important feature in the continued recruitment and enrollment efforts 9 In order to prevent further treatment group customers without installations action is in place to preclude ineligible
32. make adjustments to my temperature settings when coming home early or late My initial pre programmed settings have kept me comfortable like the messages receive when I make temperature adjustments The thermostat iPhone and website portal have made me more aware of how much energy use in ne eel ow 18 If you have any comments about the ratings you just provided please note them here Please be as detailed as possible open end and not required 19 Have you had any problems connecting with or controlling your thermostat remotely a Yes b No 20 if yes to prior question Please describe the problems you had Please be as specific and detailed as possible open end and not required 21 Did you need to contact customer service for support at any point a Yes b No 22 if yes to prior question Please describe why you contacted customer service Please be as specific and detailed as possible open end and not required 23 if yes to question 21 Did you feel that your customer service experience was positive Using a 10 point scale would you say you had a positive customer experience 10 or a negative customer experience 1 1 2 3 4 5 6 7 8 9 10 Negative customer Positive customer experience experience 24 if Q14 is not 8 10 Why did you give a rating of insert rating from above for your customer service experience Please be as specific and detailed as possible Optiona
33. n for this trial has not yet been recruited this report is a useful point at which to assess the characteristics of customers who have been recruited so far In this section we discuss the following issues e The location of customer recruiting and customer installation e The characteristics of our test population compared with the PG amp E residential population e The characteristics of customers who have thermostats installed and e The characteristics of the treatment group including both successful installations and customers in the treatment group where installation did not occur compared with the characteristics of the control group Figure 2 shows a map with the counties targeted for recruitment shaded in and with different colored circles to represent control customers treatment customers with installed thermostats and treatment customers where installation did not occur referred to below as uninstalled Note that due to strong geographic clustering many of the control group customer circles are blocked from view by treatment group customer circles Also three customers in the trial live in areas too far away to include in the map By far the greatest concentration of customers in the trial is in Contra Costa county PG amp E s Emerging Technologies Program FIGURE 2 LOCATIONS OF RECRUITING AND TEST GROUPS Location of Treatment Control and Installed Customers Legend o Treatment Uninstalled e Treatment Insta
34. ng Enrollment Process 1 0 1 2 6 4 9 30 32 60 87 for Trial Thermostat 0 1 2 3 9 8 10 29 38 45 8 3 iPhone app 1 1 2 3 4 12 14 29 31 47 8 3 Website portal 0 1 0 0 9 6 6 14 17 9 7 7 Receiving messages from iPhone amp website 9 4 8 3 29 19 16 11 18 28 6 6 Sieh a4 1 1 2 2 2 a 3f2sa4 5 8 customer service Customers were also asked to rate the helpfulness of the messaging the thermostat manual and the material introducing them to the iPhone app and website Table 4 summarizes the response to questions similar to those in Table 3 but rated on a scale from 1 to 10 with 1 meaning not helpful and 10 meaning helpful TABLE 4 FREQUENCY OF RESPONSES TO HELPFULNESS QUESTIONS AND AVERAGE RESPONSE Not Helpful Rating 1 10 Helpful Feature 10 Average Rating Messages from iPhone amp Website Thermostat Manual 12 2 6 2 30 18 10 21 14 20 6 4 iPhone App amp Website Introduction 11 2 7 3 20 19 19 17 21 26 6 8 5 11 34 17 9 11 30 21 34 7 5 Among the support materials customers saw the thermostat manual received lower ratings for helpfulness than did the introductory materials for the iPhone app and website or the messages they received For all but three of the overall impression and helpfulness questions respondents were asked to provide comments on why they gave the score they did There were several useful comme
35. nts which are discussed next 18 PG amp E s Emerging Technologies Program ENROLLMENT PROC ESS Approximately 90 of the comments were from people who rated the enrollment process 7 or higher The majority of responses therefore reflect suggestions more than complaints because overall satisfaction was high Summarized below are some of the more frequently occurring comments e Inthe eligibility process the requirements were unclear e Several customers disliked the wait between applying and knowing if they were eligible e Several customers mentioned that the scheduling website went down and e Several customers had trouble finding the time for installation However the majority of negative comments cited small technical hiccups that made the process tedious Examples include not finding account numbers having to call again or go online after signing up to confirm enrollment no wi fi at the booth and forgetting passwords The majority of positive comments cited the friendly and helpful support staff and technicians at the booths over the phone and during installation Many people also commented on how simple and quick the process was at both the booth and on the website IPHONE APP The majority of comments were from customers who gave the app a rating greater than 5 The most frequently recurring comments had to do with how much customers enjoyed being able to remotely control their thermostats as well as the ease of use
36. ould recommend the thermostat and app to a friend Finally the main findings regarding customer characteristics include e Customers in the trial have substantially higher energy usage and tend to have higher incomes than the broader PG amp E population e Customers with successful thermostat installations have similar characteristics to the broader treatment and control groups indicating that installation failure is mainly a random event and e Treatment group customers are comparable to control group customers across all observable variables indicating a well implemented experiment thus far PROJECT RECOMMENDATIONS Recommendations from the trial so far are to continue to recruit until the test group reaches a size appropriate for estimating energy savings This will entail a minimum of 500 installed thermostats Recruitment would be easier if an app was available for smart phones rather than only iPhones and if customers were offered a consolation prize if they were randomly assigned to the control group Every effort should be made to reduce failed installations at treatment group premises Note that action is being taken to improve recruitment efforts An Android version of the smartphone app is available for future customers recruited to participate in the trial Also while many of the treatment group customers whose installation could not occur were ineligible to participate the ineligibility was not determined until an insta
37. p and customers who have thermostats installed are quite small indicating that missed installation opportunities may be more or less random by observable characteristics TABLE 9 CUSTOMER CHARACTERISTICS ae Test Group Residences with Category poe Treatment amp Thermostat Pectridty Control Installed Population Zone P 4 0 0 ZoneQ 0 0 0 Zone R 11 0 0 Zone S 16 34 29 Zone T 24 0 1 Zone V 1 0 0 Zone W 5 0 0 Zone X 38 65 70 Zone Y of Group 1 0 0 Zone Z 0 0 0 lt 30k 39 15 13 30k 50k 10 5 5 50k S80k 21 20 22 gt 80k 30 59 60 CARE 27 10 9 Rent 6 1 0 Pool 12 29 31 Residents sch cdinata 2 6 3 1 3 1 ite hag ane 1954 2002 2004 Average year Year Built built i ae ee 1973 Average head of Adult Age household age aa e a Figure 3 compares one year of pre trial monthly usage values between the same three groups Again the test group is quite similar to the group with successful thermostat installations Both have much higher average usage than the general population which is 26 PG amp E s Emerging Technologies Program expected because recruitment has focused on areas where customers are expected to use relatively more electricity and gas for climate control FIGURE 3 CUSTOMER MONTHLY USAGE PG amp E Residential Poppulation Control amp
38. p is easy to use 1 O 3 3 11 2 15 17 27 66 8 5 The website portal is easy to use 6 15 3 3 39 12 11 17 18 31 6 8 The iPhone app and web portal provide useful information lm glad I signed up for the program fe ee A 2 Se BE ee 87 8 9 make adjustments to my temperature settings when coming home early or late My initial pre programmed settings have 4 4ai9 61 15113116 23125 30 72 kept me comfortable like the messages receive when make temperature adjustments The thermostat iPhone and website portal have made me more aware of how much 6 8 6 7 14 11 19 14 17 43 7 1 energy use w O A A N wo 19 18 31 50 8 1 12 4 5 0 15 5 12 23 21 48 7 4 12 94 5 299 15 16 16 30 6 5 Overall responses were positive No average rating fell below 6 5 Messages after temperature adjustments and ease of use of the website portal received the lowest rankings However even in those cases many customers gave quite high ratings All participants in the survey responded to all of the questions Receiving particularly high average ratings were the questions about whether the combined solution gives the customer greater control over their energy usage whether they would recommend the thermostat to a friend and whether they are glad they signed up for the program This is encouraging from a customer experience point of view Also the most common ans
39. pen end and not required 9 About how often have you tried using the website portal to control your thermostat a More than 16 times b 6 15 times c 2 5 times d Once e never 10 if not e to prior question We are interested in your impression of the website portal Again using a 10 point scale would you say like the website portal very much 10 or do not like the website portal 1 1 2 3 4 5 6 7 8 9 10 Do not like the website Like the website portal portal very much 11 if prior question answered Why did you give a rating of insert rating from above for the website portal Please be as specific and detailed as possible open end and not required 12 The iPhone app and website portal provide messages intended to keep you better informed about your temperature choices and how they will affect your gas and electric bill Again using a 10 point scale would you say the messages were very helpful to make efficient choices 10 or were not helpful to make efficient choices 1 1 2 3 4 5 6 7 8 9 10 Messages were not Messages were very helpful helpful 13 if prior question answered Why did you give a rating of insert rating from above for whether the messages helped you make efficient choices Please be as specific and detailed as possible open end and not required 14 Also ona 10 point scale would you say you liked receiving the messa
40. rgy savings and to understand the customer experience with the thermostat Energy savings measurements will be accomplished through statistical analysis of monthly usage data after more customers have been recruited and sufficient time has passed for savings to be measured which capture a heating and cooling season for customers The primary objectives of the data evaluation in this report are to understand the current test population to document the recruitment and installation process so far and to determine any improvements that can be made to the process or to the customer experience Understanding the customer experience and related issues is done through evaluation of the recruitment and installation process and through surveys of customer experience with the thermostat and associated apps The data from this wave of the project consists of e The lists of customers who were recruited into the trial those assigned to the treatment and control groups and those who have had thermostats successfully installed thus far e Data on the installation success rate of customers recruited into the treatment group and the reasons for installation failure 10 PG amp E s Emerging Technologies Program e Data on basic demographic characteristics of customers visited by Honeywell installers which was gathered using a short survey form filled out by customers at installation e Customer satisfaction results from a survey of customers done a
41. t The survey took place in mid November after customers had the thermostat installed for 2 3 months The survey achieved a 52 response rate with 145 customers responding out of a pool of 276 Survey results indicate a positive response to the thermostat and installation process so far Customers particularly enjoy being able to remotely control their thermostat and most would recommend the thermostat to a friend PG amp E s Emerging Technologies Program PROJECT GOAL The goal of the trial is to test the customer experience and energy savings from the thermostat and behavioral software solution The thermostat and associated software are intended to bring about several behavioral changes in customers with respect to how they heat and cool their homes including e Driving a higher percentage of customers to program their thermostats with a custom schedule e Motivating customers to set a more efficient temperature when home e Increasing set back temperatures i e automating systems when customers are away from home and e Expanding the length of the set back period PROJECT DESCRIPTION The evaluation of the trial up to this point consists of three main pieces The first piece consists of an evaluation of the recruitment and installation process The second piece is an analysis of the characteristics of customers recruited into the trial The third piece is a summary of results of a survey done on customers with installed de
42. t installation and e Results from an email survey of installed customers performed by Population Research Systems PRS the market research arm of Freeman Sullivan amp Co The customer lists are used to provide comparisons of the installed and non installed customers within the treatment group the treatment and control groups and the installed and treatment groups to the broader PG amp E residential population Each of these comparisons is useful for understanding the future EM amp V effort and also for assessing the potential conservation effect of offering this technology is to the broader PG amp E population The survey data is evaluated through summary tables of survey responses as well as qualitative judgments of comments made to open ended survey responses TEC HNOLOGY PRODUCT EVALUATION The evaluation of the thermostat in the current wave of the trial is being done by PG amp E customers in a cluster of areas around Yolo Solana Contra Costa and San J oaquin counties The areas were chosen based on the expectation that they would have a high number of eligible and interested customers who could realize measurable energy savings from using the device The criteria that PG amp E customers had to satisfy in order to take part in the trial were that they e Had to receive electric and gas service from PG amp E e Reside in select zip codes e Have operational central heating and air conditioning e Have an iPhone e Own a s
43. t participants overall impressions of aspects of enrollment and features of the device and supporting software Table 3 summarizes the responses to those questions It shows the tally of the number of times respondents gave each rating on a scale from 1 to 10 with 1 meaning Negative Experience and 10 meaning Positive Experience Overall the enrollment process received high ratings with an average of 8 7 as did the thermostat itself and the iPhone app which both received averages of 8 3 It is important to note that the high ratings for the enrollment process are probably due at least in part to the fact that the survey was conducted only among customers who received the thermostat Customers who were placed in the control group also took part in the enrollment process and they may have less positive feelings about it but they were not surveyed The website portal messaging and customer experience with customer service received lower ratings with the customer experience with customer service receiving an average rating of 5 8 17 PG amp E s Emerging Technologies Program Note that fewer customers answered the question about the website portal and about customer service because customers must have experienced those aspects of the trial to answer those questions TABLE 3 FREQUENCY OF RESPONSES TO OVERALL IMPRESSION QUESTIONS AND AVERAGE RESPONSE Negative Experience Rating 1 10 Positive Experience Feature Rati
44. tant Overall the ability to remotely control the thermostat was the most important reason followed saving energy and saving money Of the customers that specified an additional reason under the option Other the three most popular reasons were Connectivity by iPhone It was free and Similarity to NEST 16 PG amp E s Emerging Technologies Program TABLE 2 FREQUENCY OF RESPONSES TO MOTIVATION FOR PARTICIPATION QUESTIONS AND AVERAGE RESPONSE Very Somewhat Somewhat Completely ace Reasons Important Important Unimportant Unimportant Sie 1 2 3 4 Attractiveness of 25 57 36 27 24 the thermostat Usefulness of the information provided by 80 49 11 5 1 6 the thermostat The potential for energy 109 34 1 1 13 savings Ability to remotely monitor and control the 130 11 4 0 1 1 thermostat My existing thermostat 1 1 5 138 3 9 was broken My existing thermostat 5 8 14 118 37 was obsolete The product is new and 40 56 22 27 2 2 different Saving money 97 41 4 3 1 4 Other Specify 18 4 0 1 1 3 METHOD OF ENROLLMENT Out of the 145 survey participants 84 were enrolled at a booth with the help of a staff member and 14 enrolled online Two individuals started enrollment at the booth but finished enrollment online One person s response was incomprehensible OVERALL IMPRESSION AND HELPFULNESS RESPONSES AND COMMENTS The survey contained several questions abou
45. tats research shows that the majority of customers do not know how to properly set their thermostats This suggests that there may be substantial savings available by making it easier for customers to set their thermostats and by providing reminders when it might be advantageous for them to change their settings The thermostat and software offered in this trial are meant to address this possibility by providing an intuitive programming interface and messaging each delivered through an app on the customer s iPhone or PC 2 See Findings from the Alpha Test of the Opower Honeywell Thermostat Pilot ET Project Number ET11PGE1072 gt California Energy Commission California Statewide Residential Appliance Saturation Study Final Report Executive Summary 2010 4 eQuest Building Energy Use and Cost Analysis Software developed by James J Hirsch amp Associates JJH version 3 64 was the latest release http www doe2 com 5 California Energy Commission California Statewide Residential Appliance Saturation Study Final Report Executive Summary 2010 PG amp E s Emerging Technologies Program EMERGING TECHNOLOGY PRODUCT The solution is a combination of a Honeywell internet enabled thermostat with Opower behavioral solution software which includes normative messaging and recommended schedules through an iPhone application referred to as an app The thermostat can also be programmed through a web application on a par
46. ticipant s PC Through this messaging the solution keeps customers better informed about their energy choices and the cost ramifications of those choices The solution design is intended to make it easy for customers to act and by using psychological mechanisms and consistent encouragement and feedback to motivate customers to reduce energy consumption The product in the home works as a standard programmable thermostat The innovation is the simplicity of its programming process and its ability to interact with Opower software The software nudges customers towards using less HVAC energy by making suggestions The suggestions alert customers to times when the customer could save energy by allowing the home to heat up more in the summer or cool down more in the winter The thermostat would replace either a conventional non programmable thermostat or a programmable thermostat The primary market barrier currently is that the customer must have an iPhone to use with the thermostat if they want to take full advantage of the mobile control Opower has since developed the software that will allow the thermostat to work with other types of smart phones specifically Android which will be available to remaining customers recruited for the trial Though a smart phone will always be required for mobile control a web interface is also available for remote access and control ASSESSMENT O Bj ECTIVES The objectives of this trial are to measure ene
47. time arrival for appointment 9 4 Appearance badge uniform presentable 9 5 Courtesy and professionalism 9 7 Willingness to help 9 8 Product service program knowledge 9 3 Preparedness had needed tools parts 9 7 Length of time for installation service 9 2 Quality of the installation service 9 5 Condition in which site was left 9 7 Supplied educational materials quality 8 9 PG amp E s Emerging Technologies Program Although the installation process has generally produced high satisfaction among customers in many instances customers have made appointments and not received a thermostat for a variety of reasons This occurred at 145 premises of which data on the reasons for non installation was made available for 79 instances Of these 16 were due to customers changing their mind 8 were due to incompatible HVAC equipment at the customer premise 18 were due to malfunctioning HVAC equipment at the customer premise 15 were due to the customer having an incompatible router and 16 were due to the customer having more than one heating zone There were a variety of reasons for the remaining failures Additionally there have been 47 cases where the customer failed to make an installation appointment after being assigned to the treatment group These instances of non installation present a problem from the standpoint of measuring energy savings Maintaining a valid experimental design is crucial for accurate estimates and this requir
48. tually exclusive For example some customers said things like l disliked them most of the time but sometimes they were useful Those that disliked the message said that the messages were either annoying provided more information than their interest level or provided redundant information MESSAGES FROM VVEBSITE AND IPHONE HELPFUL OR UNHELPFUL This section received very similar responses to those above most likely due to their similar subject as well as their proximity on the survey 77 of survey participants responded to this question 35 of respondents felt that the messages helped Customers especially found useful features such as being able to compare usage to those around them getting notifications when devices were left on and being told approximate savings accrued from adopting new temperature ranges Similar to the above questions 25 of responders said they did not receive any messages Here unlike the previous section customers gave their opinions on why they found the messages to be unhelpful Out of all the responses 16 found the information to be redundant 5 felt that the information needed to be customized 10 found it interesting but were not willing to forgo comfort for savings 6 were not interested in the messages and or found them annoying and 5 had no opinion CUSTOMER SERVICE This response section was different from the others in that it asked only customers who had given ratings less
49. unning continuously for 2 8 or 24 hours They are given the option to switch to an Auto setting that will run the fan more efficiently Lastly customers may be notified that their thermostat is set to Away but that someone has adjusted their home thermostat No other options are given at that point In early 2012 PG amp E conducted an Alpha test of the project documented in a previous report in which 17 PG amp E employees were recruited and had the thermostat installed at their home That test went relatively smoothly and the feedback from those employees was used to inform the planning for this wave of the trial The two main goals for this wave of the trial are to develop an understanding of e Energy savings potential from reducing residential heating ventilation and air conditioning HVAC consumption through behavioral changes and e Residential consumer preferences and attitudes towards enabling technologies HVAC consumption is targeted for this project because it represents approximately 9 of electric consumption and 37 of gas consumption for residential customers in California However the percentage of total electric and gas consumption can vary drastically depending on the climate zone HVAC consumption represents 30 of total electric consumption in several of PG amp E s climate zones but can range as high as 45 for 2 story homes in specific climate zones 4 While many customers have programmable thermos
50. vices A future report on this trial will contain energy savings estimates The results of the trial thus far are not focused on energy savings or cost savings They are focused on understanding the recruitment and installation process and customer experience thus far PROJECT FINDINGS RESULTS The main findings regarding the recruitment and installation process include e Customers with a successfully installed thermostat have a good impression of the recruitment and installation process e There are anecdotal reports of customer dissatisfaction at being assigned to the control group and e The number of treatment group customers without a thermostat installed is a problem that could compromise the precision of energy savings estimates when the impact evaluation is conducted once the trial is fully enrolled PG amp E s Emerging Technologies Program The key findings from the customer survey were e Most respondents were attracted to the trial by the functionality of the thermostat which allowed them to remotely control it e Respondents generally gave high scores to the enrollment process to the thermostat itself and to the iPhone app e Customers mentioned some minor technical problems such as the enrollment website going down and the iPhone app having trouble connecting to the thermostat and e Customers responded with a high level of agreement to the phrases I m glad signed up for the program and I w
51. wer for every question but one was strongly agree Participants were given the option of leaving comments about their ranking choices Many of the subjects covered in these comments overlapped with those from the previous two tables However some customers provided these comments on the usability of the wall mounted thermostat The buttons and printed text are too small to easily use or see and The interface is unintuitive 21 PG amp E s Emerging Technologies Program FREQUENCY OF USE AND EFFECT Respondents were asked about the frequency with which they used the iPhone app and the website portal as well as the effect of the messages and new thermostat on their energy use These responses are shown in Table 6 TABLE 6 FREQUENCY OF RESPONSES TO FREQUENCY OF USE AND EFFECT QUESTIONS did not Question Once Never receive any messages How often have you tried using the iPhone app to 77 46 18 3 1 g control your thermostat A h ften h tri ing th it bout how often have you tried using the website 9 10 29 14 83 i portal to control your thermostat How often did you proactively change your thermostat settings as a result of receiving 6 15 33 13 41 37 messages on your iPhone app or website portal Respondents used the iPhone app much more frequently than the website portal More than half of the respondents said that they had used the iPhone app more than 16 times In contrast only 6
52. wn how to use the web app SURVEY OF PARTICIPANTS Survey invitations were sent out to all 276 customers with installed thermostats on November 8 2012 Reminder emails were sent to non respondents on November 12 A total of 145 customers responded by November 20 for an overall response rate of 52 The main findings of the first survey are that e Most respondents were attracted to the trial by the functionality of the thermostat solution which allowed them to remotely control it e Respondents generally gave high scores to the enrollment process to the thermostat itself and to the iPhone app 15 PG amp E s Emerging Technologies Program e Customers mentioned some minor technical problems such as the enrollment website going down and the iPhone app having trouble connecting to the thermostat and e Customers responded with quite a high level of agreement to the phrases I m glad signed up for the program and I would recommend the thermostat and app to a friend These findings are discussed in more detail below MOTIVATION FOR PARTICIPATION AND METHOD OF ENROLLMENT MOTIVATION FOR PARTICIPATION Customers were asked to rank the importance of each potential reason for enrolling into the program presented to them Table 2 presents the summarized results It shows the tally of the number of times respondents gave each rating on a scale from 1 to 4 with 1 meaning Very Important and 4 meaning Completely Unimpor
Download Pdf Manuals
Related Search