What are the Values of Evaluation and Student Assessment for Book-It’s Literacy Programs?

We are in the process of wrapping-up, reporting out, and finalizing our evaluation work for the year. Creating summary reports for the focus groups, literacy speaker series, interviews with literacy non-profits, and interviews with potential partners for the literacy center, among other things. Most recently senior staff gathered to review the program evaluation and student learning assessment tools piloted for the education program during the 2009-2010 school year. Everyone was brought up to speed on the two areas of focus – student learning in literary analysis and theatre skills in the residency program and student engagement in the touring program. The outcomes from the data will inform programming – where to put resources, what to let go, where Book-It should get more specific about literacy, etc.

The idea of making decisions based on real numbers was exciting to everyone. The big question is funding. In order to engage in this kind of analysis requires: training teaching artists and educational staff to implement the tools, hours in the field, and hours in the office to analyze the data. At the end of the meeting each of us responded to the question, “What’s the value of program evaluation and student assessment for Book-It?” Below are some paraphrased responses to this question...

• I can use it in deciding what shows Book-It should tour.
• We may decide to drop certain program components like study guides if schools aren’t using them. Or we may revamp them so that people will use them.
• We can all be communicating the same message about Book-It and the education program.
• This can be used to leverage more funding for the program – this is the kind of information funders want–they are requesting these kinds of outcomes.
• We can use this to inform what and how we teach. Where we’re effective and where we need to get better.
• Making choices – it’s okay to let some things go.
• Working efficiently with what we have.
  

Because of the TCG grant Book-It was able engage in rigorous program and student learning evaluation. The next step is to find a way to make it sustainable without a consultant. This is one of the issues that came up at the TEAM pre-conference in Chicago – finding ways to analyze work in-house. Here’s the deal – it still takes time and money, not as much as a consultant, but Book-It and other theatres like Book-It will need to make choices and be strategic in order to realize this kind of analysis – the difference between saying something is of value and making it real through action.

--Gail Sehlhorst, Book-It

Dialogue with the Experts: Literacy and the Arts

LITERACY SPEAKER SERIES

As part of our research on literacy, we created a series of speakers in three areas that are relevant to Book-It’s work – Adult Learning, Literacy and the Arts, and Reading Comprehension. We were very fortunate to have experts in each of these areas dialogue with Book-It’s staff and board.

Some A-ha’s from Book-It:

"READING IS THINKING. It seems like a "no duh" concept now -- but I wish I had had this idea in mind a long time ago!” --Dave Quicksall, Board Member, Adapter, Actor, Teaching Artist

“I think it really clicked for me that everyone approaches reading differently but the one major connector is the arts. Whether you are visual or literary by nature, the arts is the way to put developmental skills to use.” --Samantha Cooper, Intern

ADULT LEARNING

SPEAKER: Bob Hughes, Seattle University, Associate Professor, Adult Education
TOPICS: Adult Literacy; Cultural Literacy; Gaps for adult learners and how Book-It can fill the gaps.

Bob earned his doctorate at Harvard University and joined the Adult Education program at Seattle University in fall 2007, bringing extensive higher education teaching and administrative experience. He coordinates the basic skills specialization within the Master’s degree. His expertise includes technology, family literacy, equity issues, and professional development for teachers.

LITERACY AND THE ARTS

SPEAKER: Janice Fournier, University of Washington, Research Scientist, Learning & Scholarly Technologies.
TOPICS: Arts and Literacy - how people learn in the arts; Assessment in the arts

Janice Fournier, PhD, is an educational psychologist and researcher with a special focus on human development and cognition in the arts. She has taught learning theory, assessment, and integration of the arts in the UW's teacher education program, and worked with numerous arts organizations as a consultant on program design and evaluation and faculty development.

READING COMPREHENSION

SPEAKER: Sheila Valencia, University of Washington, Professor in Curriculum & Instruction
TOPICS: Reading Comprehension; Youth and Engagement

Dr. Valencia's areas of teaching and research specializations include reading and writing instruction, literacy assessment, and professional development of teachers. She is the author of Literacy Portfolios in Action (1998) and contributing editor of Authentic Reading Assessment: Practices and Possibilities (1994). She also serves on several national and state advisory panels focused on large-scale and classroom-based reading assessment.

--Gail Sehlhorst, Book-It

Is 'Literacy' the Right Term?

The Parking Lot where we put ideas that might deserve more conversation at a later time. You can see the big ideas sprouting throughout the day from "WIA – the Workforce Investment Act" to “What is diversity” to “Is ‘literacy’ the right term.”

Jane talking about the literacy center – Not trying to teach people to read, but to give them a positive experience with a piece of literature so they can go out and find meaning in what they read.

This was toward the end...my antennas went up because we moved into generalities about Book-It’s work. I pushed for us to be specific, and to support our ideas with evidence. They said I looked more like a bull.

To close, here is an image of our original “Vision Exercise” that happened at the beginning of the grant. During a break we looked at what we imagined the literacy center to be back in November – we saw assumptions and ideas that are affirmed by the information that’s been gathered to date.

Two big things that came up at the very end of the retreat are:

1. We need to talk more about our work as a theatre. Are we trying too hard to become something else? Does our theatre inspire literacy in our community? Or does it promote something else?
2. What does inspire mean to us?
   

We are mid-way – Sha-woo...

--Gail Sehlhorst, Book-It

It's All About Rigor!

A while back we with met with the Paul G. Allen Family Foundation evaluator and program manager for some advice on evaluating Book-It’s programs around the reading behaviors of our audiences. They had great counsel and we have done everything they suggested. I just got off the phone with them again to get more counsel and the big message was...RIGOR – it’s all about rigor. Evaluation in the arts can feel like an illusive challenge – how do you quantify your artistic work and your mission?!? I have to say, while it is hard, it’s not impossible – AND getting concrete data is exciting as it informs the work – pushing us to uncomfortable places. And it’s in these states of discomfort that growth/change can occur.

This May we are finally kicking off our focus groups – the process has brought staff together to identify hypotheses regarding what we think Book-It does and the impact Book-It has on its audiences. We sent about 3300 emails out to different segments of our audience and got a huge response from people who want to participate – 400 so far.

Next steps are to turn our hypotheses into good questions for surveys and the focus groups – turns out crafting good questions is an art. We’ve piloted many things and continue to morph how we phrase questions and statements.

We’ve got a sampling of pictures of documents we’ve created to support us in these endeavors. They might provide a little visual context for the lists, statements, thoughts, and questions we’re investigating...

-- Gail Sehlhorst, Book-It

How Do You Go from Learning to Read to Reading to Learn?

A really killer A-ha!

While discussing the upcoming focus groups with Co-Artistic Director, Jane Jones, a discovery was made. Book-It staff is doing an exercise where we’re finishing our mission statement “...and to inspire our audiences to read...” examples might be “a book by the same author,” “for pleasure,” or “to see things from a different perspective.” The exercise is designed to help us develop hypotheses that we can explore in the focus groups. As I took notes on how Jane would finish our mission statement, she was making reading comprehension connections. THIS IS THE WORK! Here's the deal, we can say the Book-It Style can change/help/be a catalyst for literacy, but that means very little to learners, funders, audience members, until we are able to explicitly state how.

As we talked, I probed about her thinking & reading process that's developed because of working in the BIS - I was able to make concrete connections to reading comprehension - the process of making meaning out of words, phrases, sentences. This is the same research they did when trying to figure out how to teach people how to read. They watched, questioned, studied good readers to codify their process. These are people who naturally read well. They took these processes and they became "reading strategies" - these are what we now teach children and struggling readers of all ages when they move from learning how to read (grades K-2) to reading to learn (grades 3 and up). This is awesome and the meat of our work if we are to develop a literacy center. More will be explored with both Artistic Directors Jane and Myra Platt and perhaps other artists in the company.

And...check out the article our high school students wrote about the Book-It Page-to-Stage project. This is the project where we’ve been piloting assessment tools. It’s titled, “Think it, write it, Book-It!” – sounds a lot like a TCG grant. There are some facts in the article that are a little off - for example, it’s our 9th year not the 20th at the school – but whatever, between the pictures and article you’ll get a sense for the student work around analyzing, adapting, and performing their own poetry – all stuff we’ve been digging into because of TCG.

Think it, write it, Book-It!

-- Gail Sehlhorst, Book-It

Planning the Spring High School Residency

It feels like spring. That must mean it's time for Book-It to work out the logistics for the upcoming high school residency.

February we had an energizing meeting with the Seattle Literacy Consortium. The Consortium has nine members, all of whom are non-profit organizations that offer free or low-cost basic-skills programs to adults seeking to learn English or improve their basic literacy skills. They embraced our work and gave great guidance on strategies to support successful partnerships in the world of adult literacy.

Primarily we have been wrapping up a high school residency during which we’ve been assessing student work and evaluating our teaching practice. Right now, we are buried under piles of paper, photographs, video and audio recordings from four classes and a total of 113 students. These piles include student poems, adaptations, literary analysis worksheets, student reflections, performance images, collaborative dialogue, and assessment checklists. We are reviewing all the documents and coding themes that tell the story of the students’ experience. Then we’ll take this information and make adjustments for our spring residency where we work with five more classes at the same school.

-- Gail Sehlhorst, Book-It Repertory Theatre

What Did Book-It Learn from Interviewing the Literacy Community?

January was a busy month; we interviewed more organizations in the literacy community and gathered data and research on adult learning in reading, writing, and speaking English. The literacy community is excited about our work and the possibility of a literacy center. One of the professors we interviewed said, “It puts learning in context for adult learners. Book-It gives them a vehicle to explore, create, discuss and, if you use literature and stories of the people you’re working with, it gives them power over their learning.” The connections the literacy institutions and experts are making to Book-It are immediate – they get how our work can support their learners and the community.

We are also deep in the process of developing tools and systems to assess student learning in Book-It All Over, our arts education program. We are starting with one of our residencies that we’ve been doing for nine years. We’ve been planning with a team of teaching artists, developing and testing systems, and revising as we go. We will be able to test the systems with the same project in the spring. Then do a final revision for future projects and programs for Book-It All Over. The process will help us get specific about how the Book-It Style impacts student learning in terms of literary analysis, communication, and comprehension.

Next will be focus groups with audiences, artists, and educators.

If anyone reading has used evaluation and student learning tools/systems for their education programs that seem to work well, please share them with us. We will be sharing our work when it’s complete. This is a time for us to test methods and models and it would be so good to know what’s out there currently – why completely re-invent the wheel if we don’t have to?

-- Myra Platt, Book-It Repertory Theatre

Book-It Repertory Theatre and Literacy

Book-It Repertory Theatre is asking many questions relating to literacy as part of their A-ha! project. They have provided an Evaluation Plan that outlines their process and seeks to answer numerous questions. Their plan and story so far...

To be honest, I feel like I’m just getting started. Through this grant, I have been hired as the Literacy Assessment Director to research, develop systems, provide data, explore, question, and answer all things related to literacy and how those things intersect with the work of Book-It. The dream is to have our own home – a literacy-based theatre center. In the beginning, I spent a lot of time developing a plan that outlines exactly what we hope to do over the course of the year. We have some big ideas and lots of questions and it became clear, that we needed to stay focused – the plan helps us achieve this and is included in this posting.

First, a little background on the Book-It Style. Book-It’s mission is to transform great literature into great theatre through simple and sensitive production and to inspire our audiences to read. Book-It adapts works of literature for the stage in the signature Book-It Style which activates the narrative text through the characters in the story. You can see/hear an example on our website or go directly to: Book-It Rep Theatre's Channel

The evaluation plan is a framework I've morphed from various consulting companies. Its purpose is to: identify the key areas of investigation; develop core questions for exploration; name the ways in which those areas will be investigated and how data will be gathered; and finally to state how this knowledge will be shared with others, or, the deliverables.

So, literacy. What is it? And, what is it in relation to Book-It’s mission? During one of our interviews with Literacy Source – a local non-profit provider of adult education, I asked “What is literacy?” The Executive Director laughed at me – it’s such a big question. She told me to do some street talk. Go on corners and ask the people – it can be interpreted so many ways. Even though she was joking, I think we should do this.

The first wave of our work was to create a staff baseline. We drafted questions to be answered for ourselves and our departments. We completed a vision exercise to draw the new “space” and the programming that it will house. This helped us see the possibilities that are floating around in our individual brains. We drafted lists of all the organizations and people Book-It has partnered with over the past 20 years. These will shape who we talk to and think of as potential future partners for the space and programming.

The questions we brainstormed are rich – they are risky because they make no assumptions about the work we’re doing and they go beyond our theatre and into our culture. Here’s just a few of these questions:

1. What stimulus and at what age triggers a person to become a reader? If this stimulus is not triggered, what can shift this behavior?
2. What does Book-It mean when they say that they encourage audiences to read – read what? How much?
3. What is it about reading that keeps us engaged as citizens?

So…What is literacy? We have started to gather statistical data for our region on illiteracy rates, programs, immigrant populations. We are doing a literature review that includes the NEA’s study around reading literature/poetry. We’ll be doing interviews and focus groups with the community and literacy-based organizations.

The next wave will be to take this knowledge and hone-in on what literacy means to Book-It. We want to measure the impact of our work for the mainstage and education programs. We want to know how to use the Book-It Style intentionally within the world of reading. I think once we’re educated on all this, the lines can begin to blur. Book-It already does this in many ways, but it’s helpful to name the similarities and differences.

Actually we are entering this phase now. We completed a baseline survey of our mainstage audience in response to our first show of the season – A Confederacy of Dunces – and will refine it in upcoming shows to try to understand the impact of our work on reading behaviors of our adult audiences. We are in the process of doing program evaluation and student learning assessment in our education program, Book-It All Over, with one of our Page-to-Stage residency programs. We will be evaluating our teaching methods with an eye toward “the language of possibilities” – I believe this idea comes from Shirley Brice Heath. This is not traditional instruction. We hope to compare the average classroom with a Book-It classroom when analyzing text. We will also set student learning goals for the project, which we've never had before. Fortunately we can develop and test the methods this winter, then revise for the spring when we work with another group of students.

These questions we’re grappling with are not new, but they are so important for us to address and answer as deeply and honestly as we can if we are to transform as an organization. It’s easy to feel overwhelmed by their complexity, but the conversations they prompt among ourselves and with others in the community are so exciting that I keep chasing them, trying to pin them down and analyze them in the context of the bigger picture for Book-It.

For now, I’ll stop here and will check back in a month. Thanks for reading and if any of you out there are doing similar work or have thoughts on where we are so far, we’d love to hear them!

-- Gail Sehlhorst, Literacy Assessment Director, Book-It Repertory TheatreGail has more for us regarding the Evaluation Plan and you can view the actual document here:

BOOK-IT EVALUATION PLAN 09-10

The evaluation plan is a framework I've morphed from various consulting companies. Its purpose is to: identify the key areas of investigation; develop core questions for exploration; name the ways in which those areas will be investigated and how data will be gathered; and finally to state how this knowledge will be shared with others, or, the deliverables.