Below are the slides from a presentation I am giving today at Colorado Association of School Executives annual conference in Breckenridge, CO. My goal is to get the point across that we don't need more assessments until we are effective at using the data we do have. The point is not that additional assessments aren't needed to better diagnose and progress monitor, but rather that we need to get past the idea that data in and of themselves will change instruction. Data only talk back if you listening and learning to listen is a process. The point to the executives (administrators) at this conference is that we need to make it easy for our teachers to listen. Check out the slides:
Thursday, August 02, 2007
Balanced Scorecard Metrics
Adams County School District 14 is working to develop a Balanced Scorecard that measures all perspectives (Financial, Learning and Growth, Internal Processes, and Customer/Student Achievement). We have created a strategy map (see the PowerPoint slide below) and a mock-up of our BSC (see the excel below). We are still finalizing the metrics and how they will appear on the scorecard. Our next steps are as follows: (1) include draft data (you will see the draft does not have data yet), (2) test the value of the card with data, (3) determine effective ways to measure objectives that do not yet have metrics. We are open to input.
In our development of the BSC, which is as much a framework for our strategy as it is a reporting tool, we first developed the strategy map. The strategy map frames the interrelationship between perspectives and objectives. In addition, the strategy map is communicating what is important to the district. What becomes problematic is that everything that is important is not easily measurable. For example, one of the things that we have identified as important is to "improve student products." We think this is important because in the 21st Century it is not enough for students to complete the requirements of the traditional school system (e.g. worksheets). Students need to be demonstrating their skills at 21st Century skills like presenting, analyzing, and communicating to and with the wider global community. That said, we haven't the slightest idea how we are going to measure the improvement in student products. What is key is that we still put the objective on the BSC. Even if we haven't figured it out we send the message that this or will be important to the district achieving its mission.
In our development of the BSC, which is as much a framework for our strategy as it is a reporting tool, we first developed the strategy map. The strategy map frames the interrelationship between perspectives and objectives. In addition, the strategy map is communicating what is important to the district. What becomes problematic is that everything that is important is not easily measurable. For example, one of the things that we have identified as important is to "improve student products." We think this is important because in the 21st Century it is not enough for students to complete the requirements of the traditional school system (e.g. worksheets). Students need to be demonstrating their skills at 21st Century skills like presenting, analyzing, and communicating to and with the wider global community. That said, we haven't the slightest idea how we are going to measure the improvement in student products. What is key is that we still put the objective on the BSC. Even if we haven't figured it out we send the message that this or will be important to the district achieving its mission.
Wednesday, June 20, 2007
Building a Balanced Scorecard in K-12
The Balanced Scorecard is a method for measuring progress towards systemic strategic objectives. The BSC was created by two Harvard B-School Professors that argued that businesses spent too much energy focusing on a large number of metrics only related to the bottom line. Many companies, they argued, did not measure customer satisfaction, operational efficiency, or the preparedness of their employees to address their strategic objectives. Kaplan and Norton argued that companies needed to understand the relationship between well-trained employees, operational efficiency, happy customers, and meeting financial targets. The BSC is the resulting framework.
The BSC includes measurable objectives related to staff development, operations, customers, and finances (profit). The BSC is then used to continuously tracked the performance of your organization in achieving these objectives. The BSC has helped many businesses better understand their progress. In fact, half of the Fortune 500 use the BSC.
Public schools are frequently wary of adopting methods from the business sector and the BSC is no exception. However, as a framework it makes perfect sense and is easily adaptable. Move finance to the bottom of the hierarchy and it makes sense. Customers (students) become the focus of the organization rather than Financial. The stream of questions is as follows: How do we manage budgets to ensure that we can provide great professional development? What do we do in professional development to ensure great teaching? What does great teaching look like to guarantee students (customers) thrive in our system?
Adams 14 is working hard to create own BSC now for the educational division.
Here are some school districts that use the BSC:
Tuesday, March 20, 2007
Measuring What Matters Part II
A new report will be released in California regarding the state of education. The report, prepared by 30 "experts" in the field, cites difficulties in sharing data, too many regulations and requirements for leaders, and ineffective methods for identifying effective teachers.
In a previous post I argued that we should be working to "measure what matters". In other words, if we want to know whether a teacher is effective we need to develop an example of "effective" and tools for determining whether the teacher is achieving that example. Teachers and leaders need to know where teachers on a scale (or rubric) and they need to know what to do next to improve. Years of service apparently can differentiate a 1st year teacher from a 5th year teacher, but after that there is little difference between a 5th year teacher and a 15th year teacher. Instead of seeing our longest serving teachers as the most skilled, we should create measures to see our most skilled teachers as our most skilled teachers.
In a previous post I argued that we should be working to "measure what matters". In other words, if we want to know whether a teacher is effective we need to develop an example of "effective" and tools for determining whether the teacher is achieving that example. Teachers and leaders need to know where teachers on a scale (or rubric) and they need to know what to do next to improve. Years of service apparently can differentiate a 1st year teacher from a 5th year teacher, but after that there is little difference between a 5th year teacher and a 15th year teacher. Instead of seeing our longest serving teachers as the most skilled, we should create measures to see our most skilled teachers as our most skilled teachers.
Monday, March 19, 2007
Many Eyes is Addictive Data Visualizing Tool (it's social too)
Many Eyes, a product of the IBM Visual Communication Lab, is a wicked cool tool for seeing all kinds of data in a new way. Many Eyes says their goal is "to "democratize" visualization and to enable a new social kind of data analysis". In practice this means that all data uploaded to this site are public and all visualizations are public and can be commented on (like a blog).
Many Eyes is user-friendly and addictive. I kept loading new forms of data to try different views. I loaded school district demographics over time, speeches made by Margaret Spellings, and rap lyrics from Ice Cube and TuPac.
Imagine the potential in a K-12 setting where students could be challenged to collect data (primary collection or secondary) and then these data would be shared across the world. their classmates could comment, but so too could an expert in the field they are studying. This tool not only creates opportunities to see the world in a new way (literally), but also to collaborate and understand the world more deeply (or see it in a new way metaphorically).
Check out these examples I created:
Example #1: District demographics over time. Click the image here and drill-down (using the plus signs) to the demographics by school, ethnicity, and gender.
Example #2: A tree map of two years of demographic data reveals change in color. Check out the hover over Ajax features.
Many Eyes is user-friendly and addictive. I kept loading new forms of data to try different views. I loaded school district demographics over time, speeches made by Margaret Spellings, and rap lyrics from Ice Cube and TuPac.
Imagine the potential in a K-12 setting where students could be challenged to collect data (primary collection or secondary) and then these data would be shared across the world. their classmates could comment, but so too could an expert in the field they are studying. This tool not only creates opportunities to see the world in a new way (literally), but also to collaborate and understand the world more deeply (or see it in a new way metaphorically).
Check out these examples I created:
Example #1: District demographics over time. Click the image here and drill-down (using the plus signs) to the demographics by school, ethnicity, and gender.
Example #2: A tree map of two years of demographic data reveals change in color. Check out the hover over Ajax features.
Example #3: Tag Cloud of recent testimony from Secretary of Education Margaret Spellings. Any idea what Spellings intends to focus on? Could you imagine students using this to compare the language in two poems, rap lyrics, books on the same subject in two different decades, speeches...or anything else? Imagine how engaged students would be to see the text they are analyzing come alive.
Wednesday, March 07, 2007
Great Design Works for Data Display Too
Excellent post over at Bokardo called Five Principles to Design By. In this post Joshua Porter identifies Five Principles that social web designers should keep in mind and these are easily extended to any school district or architecture designer that is thinking about data display.
First, technology serves humans. If the technology fails or the user cannot figure out how to get value from it, then the design is the problem. A simple concept that seems to be overlooked when considering how to display data to teachers. Second, design is not art. Art is to be enjoyed and design facilitates use. Third, designers do not create experiences, they create artifacts to experience. This seems to be akin to Kathy Sierra’s argument that serving our customers means that in the end it is about them kicking ass. Fourth, great design is invisible because it solves a problem and works well. We take it for granted. Fifth, simplicity is the ultimate sophistication. Distill the design to the absolute needs to solve the problem at hand, that is all.
This all applies directly to design of data systems for school districts to use. If the ultimate goal is for teachers to use the data system to analyze data and track student progress, then the design of the system must be teacher-friendly, focused on their experience, and simple enough that the user does not have to read the freakin’ manual.
First, technology serves humans. If the technology fails or the user cannot figure out how to get value from it, then the design is the problem. A simple concept that seems to be overlooked when considering how to display data to teachers. Second, design is not art. Art is to be enjoyed and design facilitates use. Third, designers do not create experiences, they create artifacts to experience. This seems to be akin to Kathy Sierra’s argument that serving our customers means that in the end it is about them kicking ass. Fourth, great design is invisible because it solves a problem and works well. We take it for granted. Fifth, simplicity is the ultimate sophistication. Distill the design to the absolute needs to solve the problem at hand, that is all.
This all applies directly to design of data systems for school districts to use. If the ultimate goal is for teachers to use the data system to analyze data and track student progress, then the design of the system must be teacher-friendly, focused on their experience, and simple enough that the user does not have to read the freakin’ manual.
Monday, February 12, 2007
Make A Film For Your Parents
Given the imprecise nature of the language we use to communicate with parents regarding their student's performance, I suggested five steps that schools or teachers can take to improve communication. However, I keep playing this issue over in my head and constantly wonder if there is a better way.
What if we were delivering the results as if we were telling a story or making a movie? Twice a week I work with elementary school students in a film club. We work with these students to transform their ideas in to a story (see the district film festival website here). In film, we use a method called storyboarding to visualize what we want to happen. In a sense teachers or schools could think through this very same process with each student for each step: (1) What is the test, when did your son/daughter take the test, and why the results are important? (2) How your student scored some comparison data (e.g. how did the rest of the state, district, or school score)? (3) How you can support your student to improve or maintain high performance? (4) When the student will test next? (5) What is the best way to contact their teacher to get more information?
Above was my picture for the first scene in the storyboard. Here is my storyboard for the first two images. The metaphor of a storyboard is great because it makes you think through the emotion and imagery that you want to create, just like a great director does with a movie. The storyboard version of talking to parents about students could result in better metaphors and descriptions of the student's actual performance. It is a deliberate method for creating a vision for the parents.
Tuesday, February 06, 2007
Mime the Results
The way you present data should slef-explanatory enough that a mime could walk most people through the reports.
At a keynote address I attended last Thursday Tom Guskey said, "If parents don't understand what we are trying to tell them it is our problem, not their problem." Tom Guskey went on to demonstrate that even when we think we are sending a straight-forward message to parents (e.g. end of term grades) there is hidden meaning and unintended messages. Guskey pointed out that grades frequently are inconsistent and include multiple dimensions (e.g. homework, attendance, participation, test perfromance). So a student who knows and understands the content could easily earn a C if they refuse to do home work. A couple of years ago we compared ACT and CSAP performance with student grades within relevant subjects. Guskey said this relationship was weak, but we found no relationship at all (for you stat nerds I believe the correlation was around 0.04). This study led to a re-examination of grading procedures and relationship to the standards.
No matter what assessment or performance related material we are sending home to parents we should be sure that we are clear about the following things: (1) What is the test, when did your son/daughter take the test, and why the results are important. (2) How your student scored some comparison data (e.g. how did the rest of the state, district, or school score). (3) How you can support your student to improve or maintain high performance. (4) When the student will test next. (5) What is the best way to contact their teacher to get more information.
How often are we successful at all of these steps?
No matter what assessment or performance related material we are sending home to parents we should be sure that we are clear about the following things: (1) What is the test, when did your son/daughter take the test, and why the results are important. (2) How your student scored some comparison data (e.g. how did the rest of the state, district, or school score). (3) How you can support your student to improve or maintain high performance. (4) When the student will test next. (5) What is the best way to contact their teacher to get more information.
How often are we successful at all of these steps?
Subscribe to:
Posts (Atom)