Tag Archives: evaluation

Fancy Doodles

23 Jul

One of my favorite parts of my still relatively new job as an evaluator is being able to tell the stories of our programs not only in words, but also in pictures. Regular readers of my blog know how much I enjoy and value sketching, doodling, and drawing as part of my work process, and I’ve enjoyed sharing my sketchnotes over the past few years here on my blog, but my new role allows me to create infograms for each newsletter produced by the UMass Center for Clinical and Translational Science. I thought I’d share some here.

When I had to report on the current work of our funded clinical scholars, I decided to highlight how a small group of people (6) can lead to much larger groups and connections and ultimately, outputs such as subsequent funding and peer-reviewed papers. Turning those facts into pictures, I came up with this:

Slide1

For those curious, I used PowerPoint to draw this graphic. The dollar signs and presentation screens are clip art, but the rest I was able to draw by hand. You can draw pretty much anything with triangles and rectangles and circles. 🙂

Next, I had to report on the progress of another group of funded researchers – our Pilot Project Program Awardees. I took the information given to me via lengthy written reports and turned it into this graphic to show the importance and value of Team Science. For this one, I tested out the infographic site, Easel.ly. It allows you to do many things via their free version.

PPPSummaryUpdateSpring2015

Most recently, the Principal Investigator for our Center wanted to know about the funding of these Pilot Projects since we began doing so, back in 2007. What could we say about this program, since we initiated it? I decided one thing worth evaluating was our return on investment. Since 2007, the UMCCTS has awarded around $5 million to fund research that promotes collaboration between basic science and clinical researchers, provides seed funding for ideas to grow, and advances translational science. What’s been the return on that investment? Turning back to PowerPoint, I created this graphic:

PPP Investment

It’s a challenge to collect and analyze the data behind these images, but in many ways the bigger challenge is to figure out which story is the one to tell and how best to tell it. It’s a skill of an accomplished evaluator, something that I can’t really call myself only 8 months into the job, but I’m happy to report that it’s both interesting and rewarding to work towards such a goal.

A Sheep in Sheep’s Clothing

28 May

[Alternate Title: The Sheep that Wags the Wolf’s Tail]

Photo Credit:  petsadviser.com

Photo Credit: petsadviser.com

A few years back, I made a deal with a friend to run the Chicago Marathon. I’d run several marathons previously, but none in a number of years. I was nowhere near the shape I wanted to be in when I made the deal and as both my friend and I were trying to be healthy, it was a win-win situation. At least it seemed that way. I trained in the winter, indoors on the treadmill. Once spring came, I moved outside. Week after week, I slogged through, never feeling like I was gaining any stamina, losing any weight, or getting any healthier. As summer came along and I started to stretch out my miles, I found myself getting pretty sick after running. I couldn’t eat anything without having serious digestive troubles. It got to the point that after running I’d stick with a smoothie and not much else. Even that didn’t always go well. I’d never had this experience in past training and kept chalking it up to being out of shape.

Then, one Friday evening I went for a 12 mile run – struggling through it as my gut rebelled against me. I finally finished, stopped by the 7-11 for a Gatorade, and drove home. After showering, I had a smoothie and settled in to watch the Red Sox. Next thing I knew, I was on the floor of my bathroom and next after that in an ambulance to the emergency room. I’d never been in an ambulance before – and that’s about the only positive I can think of regarding the experience.

After a night in the ER, tests that revealed nothing much, trips to my doc and a couple of specialists, the vascular surgeon told me that he suspected I had celiac artery compression syndrome (or median arcuate ligament syndrome). He also said, as I described my symptoms related to running with him, that he’d never heard of it being associated with exercise.

Being the medical librarian that I am, I set about searching PubMed (now that I knew some terms to search) to learn about what was going on inside of me. Mostly, I was looking for something that would link my training with this syndrome. Lo and behold, I found one. One. That was it. One case study about one individual – an elite runner who’d suffered something similar to what I was experiencing. Granted, I was hardly “elite” in my running, but the symptoms and situations described for this runner were just what kept happening to me.

I promptly sent a copy of the article to my surgeon and then, a couple of weeks later when I was wheeled into the operating room for an arteriogram to confirm his diagnosis, he said to everyone in the room, “If you have any questions, ask Sally. She’s read more about this than you have.” (Though fortunately not more than my surgeon!)

This is a long, round-about story to demonstrate a point – when it comes to evidence, a case study that resonates with you, the individual, is worth as much as any randomized control trial.

Evidence-based practice is THE term in medicine today. As noted by Stewart Donaldson, Christina Christie, and Melvin Mark in the introduction to their book, “What Counts as Credible Evidence in Applied Research and Evaluation Practice?” we live in an evidence-based society. Evidence-based medicine, evidence-based mental health, evidence-based management, evidence-based decision making, evidence-based education, evidence-based coaching, evidence-based policy, evidence-based sex education, evidence-based fill-in-any-blank are just some of the examples they list from a quick Google search of “evidence-based practice”. For those of us who have taught any course related to evidence-based practice, we know all about the EBM Pyramid and the hierarchy of quality when it comes to evidence. At the top of the pyramid sits the randomized control trial and systematic reviews. Further down, the anecdotal case study. In other words, something happening to one or two people – like me and that other runner – simply doesn’t qualify as enough evidence to state that there is any connection between exercise and celiac artery compression syndrome.

Except when it is enough. As it is/was for the two of us (and no doubt a few others).

I found myself thinking often of my personal case history and the evidence-based pyramid during a number of sessions that I attended during last week’s annual meeting of the Medical Library Association. Why? Well, mostly because I attended a lot of talks on the new roles that librarians and other professionals working within libraries, i.e. PhDs in bioinformatics, are assuming today. People are doing an awful lot of interesting things related to specialized services. I count myself in that lot. I may well be an evaluator now, but I personally think it’s simply an extension of the specialized work that I was doing in the library. But the thing that I kept noticing – and a point I raised in one of the sessions – was when, if ever, will we get past case studies related to these services? When, if ever, will we be able to say as a profession that the successful new roles and services that some libraries are offering today are roles and services that can be adopted broadly? When, if ever, will we have enough evidence that demonstrates the success is based more on the service and the role, and less upon the individual delivering it?

Watching Twitter throughout the meeting, I noticed one person tweet a picture of a slide from Bart Ragon’s (University of Virginia) presentation, “Where is My Data Scientist?” (Disclaimer: I was in a different session at the time, thus am taking Bart’s slide out of context.) The slide read, “Unless you are Kristi Holmes or Michele Tennant – Most librarians lack any of these skills.” For those less familiar with MLA, for many years, Kristi (formerly at Washington University, St. Louis, now the Library Director of Galter Health Science Library, Northwestern University) and Michele (University of Florida) were known as the two PhD biochemistry people in our midst. They were anomalies; scientists working in medical libraries. Today we have more – Jackie Wirz at Oregon Health Sciences University, Meng Li and Yibu Chen at the University of Southern California, and Tobin Magle at the University of Colorado Denver to name a few – but they still remain oddities.

When I asked the panel of Kristi, Tobin, Jackie, Meng, and Jerry Perry (former director of UC-Denver’s medical library, soon to be the same at the University of Arizona’s health sciences library) this question about case studies versus a broader body of evidence, Jackie admitted that sometimes she does wonder if people call on her because they think “Jackie can help” or if they think the bigger, “the library can help.” I don’t mean to suggest that the two are mutually exclusive, but how much are our trends towards specialized services redefining the health sciences/medical library profession, as a whole, and how much are they simply taking advantage of particular individuals and the strengths, expertise, etc. they bring to a particular library?

One of the bits of advice that Kristi Holmes offered during this session was, “Build the best library for your institution, thus what works here may not work there.” It’s hard to argue that this isn’t good advice, yet at the same time I can’t help but wonder about how well it sums up our future as a profession. It’s becoming more and more specialized, more and more individualized, and the parts aren’t easily interchangeable anymore. This can be either good or bad for us. Thinking to what Mae Jemison said in her McGovern Lecture (I wrote about this for the NAHSL blog and will share the link when it’s published), innovation is not inherently good or bad; progressive or regressive. It’s the choices that we make around our innovations, our new ideas, our new roles, and our new services that ultimately make the difference.

I’m a librarian with the title of “Evaluator” sitting in an office in the UMass Center for Clinical and Translational Science. Jackie Wirz is a PhD biochemist and molecular biologist with the title “Biomedical Research Specialist” and an office in the OHSU library. What do we have in common? What do we do that can be teased out, taught to others, and adopted by other libraries, centers, or institutions? To me, that’s a BIG question and something worth continuing to try and answer/address as our profession continues to redefine itself for the future (and now).

Turning the Tables

27 Mar

Earlier today, a very nice first year medical student came by my office and apologetically asked me if I could tell her where the offices of our Institutional Review Board people are. I don’t work in the library anymore, you might recall, and now my office sits in a maze of other offices on the 7th (top) floor of the ambulatory care center of our medical complex and medical school. It’s not a place that anyone wanders past. Mine is not an office that someone might simply pop in for a visit. It’s out of the way. And as I said to my former colleagues in the Library and my friends on Facebook, I admittedly miss such interruptions. Not all of the time, mind you, but one of the greatest joys that I knew as a librarian was simply answering someone’s question and / or helping them in some way that made their day better. The student was SO grateful when I got up and walked her around the corner to the IRB folks. She thanked me several times. It was a flashback to those days of yore … oh, 3 months ago or so.

For ten years, I spent the better part of my working days answering people’s questions. Either answering them directly, looking for the answer for them, or helping them learn how to find the answer for themselves. All day long, the overwhelming majority of sentences spoken to me ended in question marks. Librarians answer questions.

Evaluators, on the other hand, ask them. This is what I’m quickly learning. We are the ones who need the answers, thus we’re the ones who ask the questions. How well did “X” intervention work? How much time was saved by implementing “Y” into the process? What does “Z” do for you that no other letter of the alphabet ever did for you before?

But I’ve also learned that there’s a speed bump; the researchers and clinicians and other users of the resources and services that I’m evaluating … well … they typically aren’t librarians. In other words, I don’t think that they like answering questions as much as any librarian does.

“It’s a simple survey,” I claim.

“It won’t take more than 5 minutes of your time.”

“The results will help us help YOU!”

“I’d be ever so grateful,” said Babe the Pig. (If you miss the reference, check out the movie.)

And still, getting people to answer questions is way harder than I ever imagined. I’m actually very good at talking to people, and usually pretty good at getting people to talk to me. It was one of the skills and characteristics that I honed as a librarian / informationist that I figured would be easily transferable to my new role. Not so much. At least not yet.

It could be the method – the dreaded survey. People don’t like them. Heck, I don’t like them. But in some cases it is the most appropriate and most efficient method for getting the data (answers) you need for the evaluation. I read and studied and asked about writing good questions. I worked with seasoned researchers to put my survey together. I piloted it with different groups and made all of the necessary tweaks based on the feedback I received. I picked my target audience carefully. And once I felt confident about the whole thing, I let it loose.

And then … I waited.

And waited.

And sent out a couple of reminders.

And broadened my audience.

And worked some different angles to reach people.

And waited some more.

I’m still waiting; waiting for the responses to grow to some level that will afford me some information needed to present my findings to a couple of different groups. It’s coming along, but golly it’s slow. And such a cumbersome process. Ask me a question and I’ll happily answer for you, straightaway. But waiting for others to answer me … well, the tables have turned, my friends.

A turning table playing a different tune!

A turning table playing a different tune!