On March 24, 2006, the Quality Advocates met to discuss successful approaches to data collection and utilization, and how data can be used to improve decision-making and planning. The three panelists for the session included: Raymond Lombra, Associate Dean for Administration, Research, and College Advancement, College of the Liberal Arts; Karl Newell, Associate Dean for Research and Graduate Education, College of Health and Human Development; and, Lisa Shibley, Institutional Research and Assessment Officer, Penn State Berks. Michael Dooris, Director, Planning and Research and Assessment, Office of Planning and Institutional Assessment moderated the session.
The presenters identified three main purposes for collecting data in their colleges or areas. One reason the colleges collect and analyze data is to share the data with department heads for benchmarking purposes. Both Lombra and Newell reported their colleges provide admissions, enrollment or other data to their departments, much of the time with very little interpretation of the data. This strategy provides department heads with actual numbers to compare the performance of their unit with others within the colleges. As an example, Lombra mentioned that sharing admissions and enrollment data leads department heads to start discussion among themselves about the differences in the data and what practices others are using to achieve their results.
A second purpose of data management is to inform strategic planning and support planning for new programs. For example, Shibley reported that her work in using data to develop a retention plan for Penn State Berks was directly tied to the goals of their strategic plan. This was also the case for the College of the Liberal Arts, which looks at indicators such as placement rates of graduates and workload and capacity data in measuring progress against strategic planning goals.
A third use for data is in decision-making. With the complexity of the University and the colleges within it, deans and other administrators need some basis upon which to set goals, strategies, and priorities. Having data that portray the individual areas within colleges or departments gives decision-makers an idea of what is actually happening within these areas. Of course, units and departments vary and administrators and others need to recognize the reasons for the differences in the data and be sensitive to this in their analysis of the data. Sharing the data with faculty and staff also is important for gaining support for decision-making. Shibley cited a recent example in which the National Survey of Student Engagement, administered to students at the campus, found a low level on students reading on their own. In a special initiative, faculty selected a book which was distributed to first-time students and a common-year reading program was started. Pre- and post-tests were conducted to determine the effect of the program and the results were shared with faculty.
Some of the methodological problems that deans, department heads and others face in developing information for academic management reflect the same issues faced by academic researchers. The chief concern is that the measures have face validity. Newell pointed out that some questions are easy to measure, such as, What is the enrollment in the college?, while others are harder, e.g., What is the impact of faculty research? Developing measures that portray what is actually happening and measure what they intend to measure is a more difficult process for the latter question. In addition, two other issues which are also faced by academic researchers are understanding the parameters and nature of the data and being able to make valid interpretations from it. Newell cited the example of official enrollment data reported by the University. This data are drawn University-wide on a single day and may conflict with what department heads, who may track enrollment on a weekly basis, know as their own enrollment numbers.
In encouraging a culture which uses data for decision-making, deans, department heads and others, must assess whether the data provided meet current needs and the necessity of providing additional data. In some instances, data which was once considered essential may no longer provide the colleges and other academic units useful information. Given limited resources available for data collection and analysis and the amount of time needed to shape data into meaningful information, Lombra suggested that it is imperative to continually assess what information is needed, and whether traditional reports could be ended without hindering management practices. Shibley also related that as more information was shared at Penn State Berks, a culture of data awareness arose. However, the data raised additional questions, and in some instances, rather than making decisions based on the data originally provided, administrators, faculty and staff wanted to explore other topics. Shibley worked with campus personnel to encourage them to analyze and use the initial data before exploring other issues.
In collecting and compiling data, the presenters offered some practical advice.
- Much data currently already exist in accessible sources. These sources include the data warehouse, EIS, IBIS, ISIS, and survey data, such as the student satisfaction survey.
- Make constituents within a unit a partner, rather than an adversary, in the data collection process. The data collected should be useful and meaningful to the constituents.
- Share data with others to develop a culture of using data for decision making. All three presenters suggested that as unit heads start to become comfortable with shared data, they begin to think about their individual data needs and to develop their own performance indicators.
- Identify what data are important and focus on using that data appropriately in decision-making. Colleges and other units tend to collect everything within reach, but this takes time and money and may diminish the usefulness of the final reports.
- Continuously assess the process for collecting, analyzing and providing data to make sure that the information meets your needs and the needs of others, and to ensure that others understand the information they receive.
The Quality Advocates Network meets several times each semester to share ideas and examples of improvement and change. To join the Quality Advocates Network mailing list or to learn more about the meetings scheduled, contact the staff at firstname.lastname@example.org.
The Quality Advocates Network is open to all Penn State faculty, staff, administrators, and students.