Archive | Evaluation RSS feed for this section

Alternative Metrics ARE Common Metrics

3 Mar

I few weeks back, I was invited by the good folks at Altmetric to take part in a webinar to discuss my use of alternative metrics in my work as an evaluator for the UMass Center for Clinical & Translational Science. The webinar is available online, but for those who might want to see my slides and read the transcript from my part, here you go:

Slide01

Slide02I thought that I’d start with an overview of what’s happening regarding evaluation from the National perspective, since NCATS, the National Center for Advancing Translational Sciences, as the overseer of the CTSA program, steers the ship, so to speak. For those unaware, NCATS is the Center within NIH that oversees the 62 CTSA programs across the country.

The Common Metrics Initiative is a fairly new – or I should say re-newed/re-tooled – working group coordinated by some of the Principal Investigators from the CTSA sites. FYI, the proper jargon for a site is now, hub. So when you see the word “hub,” it refers to an individual CTSA site, such as the UMass Center for Clinical & Translational Science, where I work. Consortium refers to all of the sites, as a whole.

The Common Metrics Initiative came about in an effort to better measure, evaluate, and disseminate the impact of translational science and the concepts behind it. If you think about it, the idea of translational science is that by eliminating some of the barriers and obstacles that exist between biomedical research, clinical research, and clinical practice, discoveries that improve health will move from the lab bench to the bedside, i.e. patient care, faster. The questions of how we measure the truth of this idea is what’s behind establishing a set of common metrics – a uniform, standard set of metrics that measure the speed, efficiency, and quality of this large practice called translational science.

With multiple centers, an infinite amount of programs and research projects, countless individuals involved as researchers, clinicians, students, subjects… you can easily imagine how difficult it is to come up with a common set of metrics that everyone will collect and analyze. But it’s certainly an important thing to do, not only so that we can evaluate our respective individual hubs, but also so that we can compare across hubs.

Briefly, there are four key areas that have been identified as targets for the implementation of common metrics – workforce development (this involves training opportunities for individuals to learn, among other things, how to conduct clinical research); resources and services of each CTSA site; the collective impact of all the programs, functions, and such of an individual site; and the larger CTSA consortium as a whole.

Slide03For today’s talk, I’m going to focus on the area where alternative metrics are most useful. NCATS defines the different resources and services each hub offers as the following:

  • Biomedical informatics
  • Biostatistics, Epidemiology, Research Design and Ethics (commonly known as “the BERD”)
  • Pilot project funding
  • Regulatory Knowledge and Support
  • Community Engagement in Research
  • And Navigational Support – how well those administering hubs connect people to the resources and services that they need.

Slide04Further, I want to focus on the first three bits within this area, BMI, the BERD, and Pilot Funding.

Slide05As the evaluator for the UMCCTS, my job is basically all about answering questions. It’s a good thing that I was a librarian already, since answering questions is the librarians forte. I can also say that one of the things that I love most about being a librarian is answering interesting questions, and my role now certainly offers up a few interesting questions like these:

How effective are our resources and services – from bioinformatics to the parts of the BERD – in contributing to translational research?

When we give people money for pilot research, how well does this research then generate funding for further research? And then, what’s the impact of that research? How is it transforming health care practice and, ultimately, health?

And then the big elephant in the room, not mentioned on this slide, how do we go about answering these questions? The idea of identifying and analyzing a core set of common metrics is one attempt, but what should those metrics be?

These are big, difficult, and very interesting questions.

Slide06Of course, we start with the usual suspects. We count things. How many new projects are initiated? How many people are involved? How many trans-disciplinary collaborations are formed? How many students and new investigators are mentored and trained? How many publications result from the research done? How much new grant funding is obtained to further the work? Remember, the pilot funds offered by CTSA hubs are seed funds. They are meant to help get projects started, not fund them forever.

But what else besides these common metrics can we look to to draw a bigger picture of the success of our work? This, you guessed it, is where we look at alternative or altmetrics.

Slide07So let’s take an example. Here’s a paper authored by one of our researchers and funded, in part, through the resources and services of the UMCCTS. When we’re counting publications as a measure of success, it’s one that I can count. The other thing that I can count that’s fairly traditional is the number of citing articles. We know that this is a relatively good marker for impact – someone citing your work means that they used your work, in some way, to further their own. So the original work is having an effect. In this case, I could point out that 141 other publications needed to use this publication somehow. So we’ve got a reach, in the simplest terms, of 141 – 141 people, projects, research studies, something. This we can say from these two metrics: 1 paper, 141 citations.

But as we all know, today’s communication tools allow for much broader – and easier – dissemination of science. One of my goals, in my work, (you could see it as a challenge and/or an opportunity, too), is to help researchers and funders and other stakeholders appreciate the value of these other tools. To help them see how these tools give us a whole set of other metrics that can help us evaluate the impact of the work.

Slide08This particular paper is always a good example because you can clearly see, via the Altmetric tool, how far it’s traveled beyond the strict confines of scholarly, scientific publications. It’s also reached news outlets and social media users. It’s reached a wide cross-section of people – the general public, patients, other health care practitioners, other researchers in different disciplines. These are also important. We can argue over the level of importance, as compared to citations, but it’s difficult to ignore them – to claim that they have nothing to say when it comes to the measurable impact of this one paper.

The other reason that I like to use this particular paper as an example, besides its impressive Altmetric donut, is because the final author listed – one of the co-PIs for this work – also happens to be the PI for our CTSA site. She’s my boss. The big boss. She’s one of the PIs involved in the evaluation initiatives for NCATs. The first time that I demoed the Donut for her, she loved it. How could she not? Apart from the non-biased reaction that it’s good to see one’s work being shared, it’s also a great ego boost. Researchers, in case you don’t know, are a little bit competitive by nature. They like to see a good score, a good result, a big donut… you name it. They like it.

For those of us trying to reach the goal of bringing altmetrics into favorable light within very traditional disciplines, being able to show this type of example to your stakeholder, in this instance, my boss … it works.

Slide09So day to day, I spend a lot of time at my rock pile doing these sorts of things. I establish collections of publications, related to different groups within the UMCCTS. I maintain those collections regularly – using Collections within MyNCBI in Pubmed, or Scopus and SciVal – two tools available to me thanks to the Library of UMass Medical School. I collect data related to the common metrics outlined by NCATS, but I also collect the altmetrics. I track them all. And then I report on them all via progress reports and infographics (my latest love). It’s an ongoing – never ending – project, but it’s certainly interesting to step back from time to time and look at the big picture, the story, that all of these metrics, together, tell us.

I recently finished the final progress report for the first 5-year funding cycle of our CTSA. It really was impressive to see where we are today, in comparison to where we were just 8 or 9 years ago, when the idea of establishing a clinical and translational science center at UMass Med first took hold.

Slide10“Telling the story” is what my PI reminds me is my job. Using common and alternative metrics, I can tell the story of this one clinical research scholar who, over the past several years, has published 18 papers related to work she’s done in this program. These papers have been cited, she’s worked with many colleagues as co-authors, she’s developed a number of effective collaborations, she’s presented her work locally, regionally, and nationally, and she’s received several new grants to help her continue in her area of research. She’s also reached the public, patients, and other health care providers through multiple means.

Based on all of these metrics, I can write a pretty good story of how well this one doctor is utilizing the resources of the UMCCTS to inform practice and improve care. In a nutshell, I can tell a story of the impact of her research. If I repeat the same for each of our clinical scholars, or a group of researchers utilizing one of our research cores, or one or more of our pilot-funded projects … the story, the picture, gets bigger and, hopefully, clearer. Our Center is making a difference. That’s what we want to show. And that’s possible through the use of all of these tools and metrics.

Slide11Finally, I want and need to give a shout-out to my former colleagues and friends over at the Lamar Soutter Library here at UMass Med. I worked in the Library for 10 years before moving to the UMCCTS a little over a year ago now. It’s the work that I did in the library that first enabled me to build a relationship with our Center, and then inspired me to approach them to do the evaluation work that I do for them now. Kudos to the LSL for all of the initiatives carried out related to scholarly communications and research impact. I think together we’re helping change the environment around here and raising the level of awareness and acceptance of altmetrics.Slide12

The Art of Collaboration

12 Nov

[The following is my monthly column for the November issue of the UMCCTS newsletter.]

One of the goals of the UMCCTS is to promote and facilitate collaboration across departments and disciplines, thus effectively reducing barriers between the basic and clinical sciences, and ultimately speeding the pathway between the discovery and implementation of new treatments, therapies, and the like that improve health. One means of demonstrating collaboration is through co-authorship. The networks that develop between authors of publications give us a picture of how individuals are connected and where collaborations exist.

Social network analysis is the process of investigating social structures through the use of network and graph theories. It characterizes networked structures in terms of nodes (individual actors, people, or things within the network) and the ties or edges (relationships or interactions) that connect them. (Wikipedia, Social Network Analysis

For this month’s column, let’s look at an example of a social network analysis that shows the co-authorship relationships between members of the Division of Health Informatics and Implementation Science in the Department of Quantitative Health Sciences (QHS). QHS is one of the newest departments at UMMS, with several of the senior faculty arriving on campus only about 6 years ago. The research that the Department does in developing innovative methodologies, epidemiological research, outcomes measurement science, and biostatics is integral to the nature of clinical translational research. By examining the co-authorship relationships of members of the Health Informatics group, we get a snapshot of how well these faculty members are connecting with other departments, other disciplines, and even other institutions. In short, we see how and where collaborations have developed and thus how well the UMCCTS goal of building them is being met.

To do this analysis, we first need to identify all of the publications authored by at least one of the Division’s faculty members for the period of time that s/he has been part of the Division, as well as all of the unique co-authors associated with these papers. In doing this, I found 221 publications authored by 716 different individuals. Using Sci2, a toolset developed at Indiana University, I was able to analyze the patterns and create a visualization showing the connections between the co-authors.

Informatics Division CoAuthor Network

One thing that we clearly see is that several faculty members are prominent hubs in the network, meaning they co-author many papers with many people. Drs. Houston and Allison are the most obvious examples here. We can also see that a number of branches grow from the periphery. At the base of each of these is a faculty member from the Division (counterclockwise from upper right, Drs. Cutrona, Hogan, Shimada, Mattocks, and Yu). Finally, we note that even hubs that are less connected to the clustered middle, e.g. Drs. Yu and Pelletier, are still linked, representing the reach of the collaborative network that the Division has formed over the past years.

Tools like Sci2, Scopus, SciVal, and ISI Web of Science provide another way, i.e. a visual demonstration, of the success of our programs and the impact of the translational science being done by the members of the UMCCTS.

Sci2 Team. (2009). Science of Science (Sci2) Tool. Indiana University and SciTech Strategies, https://sci2.cns.iu.edu.

Turning the Tables

27 Mar

Earlier today, a very nice first year medical student came by my office and apologetically asked me if I could tell her where the offices of our Institutional Review Board people are. I don’t work in the library anymore, you might recall, and now my office sits in a maze of other offices on the 7th (top) floor of the ambulatory care center of our medical complex and medical school. It’s not a place that anyone wanders past. Mine is not an office that someone might simply pop in for a visit. It’s out of the way. And as I said to my former colleagues in the Library and my friends on Facebook, I admittedly miss such interruptions. Not all of the time, mind you, but one of the greatest joys that I knew as a librarian was simply answering someone’s question and / or helping them in some way that made their day better. The student was SO grateful when I got up and walked her around the corner to the IRB folks. She thanked me several times. It was a flashback to those days of yore … oh, 3 months ago or so.

For ten years, I spent the better part of my working days answering people’s questions. Either answering them directly, looking for the answer for them, or helping them learn how to find the answer for themselves. All day long, the overwhelming majority of sentences spoken to me ended in question marks. Librarians answer questions.

Evaluators, on the other hand, ask them. This is what I’m quickly learning. We are the ones who need the answers, thus we’re the ones who ask the questions. How well did “X” intervention work? How much time was saved by implementing “Y” into the process? What does “Z” do for you that no other letter of the alphabet ever did for you before?

But I’ve also learned that there’s a speed bump; the researchers and clinicians and other users of the resources and services that I’m evaluating … well … they typically aren’t librarians. In other words, I don’t think that they like answering questions as much as any librarian does.

“It’s a simple survey,” I claim.

“It won’t take more than 5 minutes of your time.”

“The results will help us help YOU!”

“I’d be ever so grateful,” said Babe the Pig. (If you miss the reference, check out the movie.)

And still, getting people to answer questions is way harder than I ever imagined. I’m actually very good at talking to people, and usually pretty good at getting people to talk to me. It was one of the skills and characteristics that I honed as a librarian / informationist that I figured would be easily transferable to my new role. Not so much. At least not yet.

It could be the method – the dreaded survey. People don’t like them. Heck, I don’t like them. But in some cases it is the most appropriate and most efficient method for getting the data (answers) you need for the evaluation. I read and studied and asked about writing good questions. I worked with seasoned researchers to put my survey together. I piloted it with different groups and made all of the necessary tweaks based on the feedback I received. I picked my target audience carefully. And once I felt confident about the whole thing, I let it loose.

And then … I waited.

And waited.

And sent out a couple of reminders.

And broadened my audience.

And worked some different angles to reach people.

And waited some more.

I’m still waiting; waiting for the responses to grow to some level that will afford me some information needed to present my findings to a couple of different groups. It’s coming along, but golly it’s slow. And such a cumbersome process. Ask me a question and I’ll happily answer for you, straightaway. But waiting for others to answer me … well, the tables have turned, my friends.

A turning table playing a different tune!

A turning table playing a different tune!

The Lost Art of Being Frugal

29 Jan

Worcester, Massachusetts got slammed by a blizzard this week, bringing out the hearty nature in all of us New Englanders. What’s a little (34″) snow to dampen our spirits? I made a big pot of chili, watched a couple of movies, read a little, and hung out with my pets while the snow flew. Then yesterday, I joined everyone else in the neighborhood in the first great dig out of the winter. It’s what you do when you live here. No complaining needed. Born out of the spirit of the Puritans that settled here, New Englanders have a reputation for hard work and frugality. Granted, it’s been some time since the days of the Pilgrims, and regional distinctions fade as we’ve become a much more migratory society over the centuries, but we still think of Southern hospitality, Midwestern friendliness, Western pioneers, and hearty New Englanders. And yesterday, we hearty folks were shoveling. 

Joseph-Siffrein Duplessis [Public domain], via Wikimedia Commons

Joseph-Siffrein Duplessis [Public domain], via Wikimedia Commons

While we praise hard work and frugality, these traits also run counter to much that Americans dream to achieve today. Retiring early, becoming a millionaire overnight, achieving fame and fortune by winning a talent contest … these are the ideas behind bestsellers and top rated television programs. We talk the talk of hard work, pulling yourself up by your bootstraps, scraping and scrapping and saving for our dreams; these are the bedrocks upon which America was built and, thus, they remain a part of our societal DNA. As one of our Founding Fathers, Benjamin Franklin, said, “The way to wealth depends on just two words, industry and frugality.” In theory. But in reality, for more reasons than we can count, they are less and less the walk of America. 

Economists speak of “frugality fatigue” as a driving force behind folks living in debt. As a species, we aren’t always very good at delayed gratification. We want what we want now and we’ve built a society that feeds this human habit in so many ways. Thus, when we also hear the popular mantra “do more with less” in our workplaces and business, it’s not something that we necessarily want to hear. It becomes a very negative thing. It wears us out, after awhile. We get stretched too thin. We simply cannot do more and more with less and less. In this sense, frugality becomes our enemy.

But is it? Was Mr. Franklin wrong? Or is there a way to look at “do more with less” that prompts something beyond stress?

According to Navi Radjou, an innovation strategist in Silicon Valley, the answer to that question is yes. In his thought-provoking TED Talk, Creative Problem-Solving in the Face of Extreme Limits, he outlines his theory of frugal innovation. In this brief talk, he gives lots of examples of people living in conditions where resources are often extremely limited, yet rather than limiting their ability to solve problems, the situation actually enhances their creativity and results in solutions that they would likely never come up with in a land of plenty. 

They can magically transform adversity into opportunity, and turn something of less value into something of high value. In other words, they mastered the art of doing more with less, which is the essence of frugal innovation.

~ Navi Radjou

We hardly need to live in abject poverty to take advantage of this idea. Librarianship, and any profession struggling with finding its footing and value in tough times, can tap into the one resource that’s common in most every situation, human ingenuity. Radjou calls it our most abundant resource. We need to find ways and create situations that foster our ingenuity. Maybe, the pressure cooker of a “do more and more with less and less” work setting can be the impetus for this. Maybe not knowing what’s coming next, not knowing where we belong, not knowing how to define and/or redefine ourselves is just the environment we need to push us towards creative solutions.

In many ways, I’m glad that I’ve entered my new role as an evaluator without a lot of traditional knowledge and background in the subject. Yes, I’ve been reading and studying up on the basics, but lacking the resource of years of experience and know-how, I find that I’m able to come up with some different thoughts and ideas and solutions that I probably wouldn’t have come up with otherwise. It’s like the team that enters the big game for the first time. They don’t know enough to know to lose.

One of the great things about evaluating the impact of clinical and translational research is that nobody really knows exactly how to do it yet. This is what I tell myself. It helps me put aside any anxiety of knowing that I don’t necessarily know what I’m doing, and sets me free to try all sorts of things in doing my job. It’s my way of making the most out of my limited resources and thus practicing frugal innovation. And that can be downright exciting. 

 

 

Making Mistakes

15 Jan

The button has been pushed and our proposal for a Clinical Translational Science Award from the National Institutes of Health is out of our hands. Let the review process begin! 

My part of the writing and work has been done for a few days now, so I’ve been spending much of the past week doing many things that I likely would have done when I started this new job, if only I hadn’t jumped into the grant writing fire. I’ve read all of the Center’s newsletters for the past few years, I’ve taken lots of notes and done a bunch of documentation related to who’s who for each of our core components. I’d drawn a lot of pictures and graphs and maps to help me understand the landscape. My informationist role in the Library prepared me well for these latter tasks, as I’d been embedded in projects and made it my job to know what was going on around the University. Still, it’s a lot of putting names to faces to departments to projects. It’s a lot to learn.

I also need to learn a lot about the role of an evaluator. I knew enough to get the job, but now that I have it, I know that I need to learn a great deal more before I’m proficient in the task(s). I’ve been reading a lot of articles from evaluation journals. I tracked down several good books on program evaluation, and a couple on evaluating biomedical and health research, in particular. I’ve subscribed to professional listservs, been reading daily blog posts by evaluators, and joined a professional organization to have access to needed resources. Lastly, I’ve been working through a few self-guided mini-courses to make sure I’ve got the basics down pat and that I understand the terminology that I’m reading elsewhere. So far, so good.

And I’m planning. It’s true that I worked with the section writers and leadership of the CCTS to develop evaluation plans for the different components of our proposal. In other words, I’ve already written a whole bunch of plans stating what I’m going to do over the next few years. That said though, there’s planning and then there’s PLANNING. The all-caps version is where I am now. I have plans to collect and track certain metrics to answer certain evaluation questions, but now I really need to plan out how I’m going to do all of that. The logistics. It’s a great challenge. It’s interesting and I’m learning a lot. I cannot complain.

A torn page from a book. I found this on the sidewalk one day during a walk. It's hung over my desk ever since.

A torn page from a book. I found this on the sidewalk one day during a walk. It’s hung over my desk ever since.

One clear thing that I’ve read – and thus learned – over and over in my study ’til now is that the practices of measuring and evaluating are continuous. You need to plan for them from the beginning and, depending upon your goal, assess at different points along the way. It’s pretty much like life in general. If you make that New Year’s Resolution to lose weight, you need to make a plan and part of that plan involves devising a means to track your progress along the way. If you want to go on a trip to Europe, you need a plan to save the money and a way to keep track of what you’ve saved, so you’ll know when you’re ready to pack your bags. 

I am a reflective person by nature. I majored in philosophy during my first time through college. I went to seminary where you hone your spiritual reflection skills well. I’ve spent time with therapists, here and there in life. I’ve been writing this blog for the past few years as a way to reflect upon and keep track of my changing roles as a librarian. For me, it’s a really helpful practice because it keeps my awareness of where I am and what I’m doing and what I’m learning at the forefront. Annual evaluations (and/or quarterly reports) don’t work for me without keeping track of things along the way. This blog helps with that.

I recently re-read something that I wrote last fall for a different blog, Hack Library School. I was interviewed, along with several other medical librarians, about our work. One of the questions asked was what advice I’d give to current students studying library science and my answer, in part, was:

Sell yourself! One of the things that I see happening in settings like mine (an academic medical school and research center) is that there is never a shortage of work for a person who can match his/her skill set to existing needs. And there are LOTS of existing needs. The key is to really know what you know how to do, know what you need to learn how to do (and learn it – ESPECIALLY if you’re weak in the sciences), and then know how to show people that what you bring is uniquely useful to them. I don’t necessarily think that this means you wait around and look for job openings in medical libraries, but that you also keep your eye on other parts of the health care system or biomedical research where what you can do fits. People looking for help often don’t think of a librarian as one who could do the job for them, but I think that’s mostly because we haven’t done the best job of selling ourselves. Know yourself, have confidence (even if you have to fake it at first), and put yourself in places that offer you opportunity.

One thing I could add to this is that developing a practice of self-reflection, evaluation, and/or tracking yourself – however you do it – will put you in a much better position to sell yourself and/or match your abilities to opportunities as they arrive. This is exactly what happened to me last November and it landed me in a great new role. My CV didn’t say a thing about being an evaluator, but I was able to map pretty much every aspect of it to the qualifications needed for the person in this post. The discipline of weekly reflection via this blog made that task easy.

We always encounter times in our lives where evaluation is forced upon us, whether it’s that mandatory annual review or a major life event. My mother-in-law is in hospice care now and nearing the end of her life. It’s a time of reflection for her and everyone in the family. “Did I live a good life?” is likely the ultimate evaluation question. You hope for the answer, “Yes” and you hope for lots of reasons to be sure of your answer, since it’s basically too late to change much. Driving home last night after visiting with her, I thought a lot about how all of these things fit together. There’s no need to wait until the program is over, until a career is over, or until a life is over to ask, “Did I do a good job?” When we plan to track, measure, reflect, and evaluate along the way, I’m fairly certain we’re better off in the end. 

January 9th – ALREADY?!

9 Jan

It’s a good thing that I didn’t make any New Year’s resolutions related to my personal writing, because I’d have to report a failure already. That said, the CTSA grant proposal that everyone has been working ’round the clock on for weeks now is very close … oh so very close … to being put to bed, which in this case means submitted. And then I’ll be able to start focusing on how to approach doing the new job that I’ve been hired to do. Up until now, I’ve only been writing what I’ll do. Next stop, figure out how to do what I said I’d do. I’ve already joined the American Evaluation Association and signed up for one of their upcoming coffee break webcasts.

An aside… I think the idea of coffee break webcasts – 30-minute weekly sessions that focus on a particular topic, led by different members of the organization – is a TERRIFIC idea. I know that I belong to a few organizations that are struggling to define and/or create the real benefits of membership and such a simple thing as a regular, free, short-and-sweet-yet-interesting webcast is just that sort of thing.

For today, I at least wanted to send up a post with a few fun things I’ve come across over the past couple days/weeks – some delayed candy canes, if you will:

  • The Spudd – it’s The Onion of medical and pharma news. Hilarious. I discovered it just this very morning, thanks to a hilarious post shared on Twitter by my friend, Dean Hendrix. 
  • How Reddit Created the World’s Largest Dialogue between Scientists and the General Public is a very good blog post by Simon Owens. I’m fascinated with scientific communication and, in particular, efforts to bring the scientific community together with the general public. We are a scientifically illiterate culture at our own peril. I love what’s happening on this online community and so I’ve set up a Reddit account and plan to follow along for awhile. 
  • Finally, for anyone curious about public health and/or epidemiology and NOT interested in returning to school ever again <hand raised>, I came across an on-demand course from the University of North Carolina at Chapel Hill. I confess that I’ve signed-up and failed at several MOOC’s, mostly because of timing. I’m really happy to find a relevant, on-demand one and hope to work through it soon. I have a feeling that doing a course on my own, at my own pace, and at my own convenience will work well for me, especially now as I juggle all of the new tasks of a new job.

Back to the grindstone here. Happy New Year to all of my readers and followers! You make blogging fun.

My 3 New Year's Resolutions for 2015. No progress yet!

My 3 New Year’s Resolutions for 2015. No progress yet!

The Great Candy Cane Caper

29 Dec

There I was merrily absorbed in the holiday spirit, running my daily Jingle Bell 5K and sharing a candy cane a day with my dear blog readers, when all of the sudden …

SCREECH!!!

It may be more appropriate to say that my two feet hit the gas pedal – VAROOOOOOM!!! – than the brakes, but regardless of how you see it, my holiday streaks were tossed off the rooftops and my blog posting has been MIA for 2+ weeks now. That said, I refuse to let 2014 close without one final post, so here goes:

If you read my last post (12/15), you know that as of that date I began a new job as a Research Evaluation Analyst for the University of Massachusetts Center for Clinical and Translational Science. I still work at the UMass Medical School campus (right down the hall from the Library), still enjoying all of the relationships I’ve built over the years. In fact, it’s those very relationships – both around the campus and in the larger academic medical library world – that helped me land this job. And while it’s immensely different in so many ways from the work that I’ve been doing for the past years as an embedded research librarian and informationist, it’s also a position that will allow me to expand on many of the skills I honed during that time.

So what am I doing now? That’s a question pretty much everyone, with the exception of my new colleagues in the UMCCTS, has asked. Right now, I’m working madly with the rest of the grant writing team, pulling everything together before the January 15th proposal deadline. The UMCCTS is funded through a Clinical and Translational Science Award from the National Institutes of Health, a 5-year grant that expires this year. As you can imagine, I came on board at one heckuva time to come on board. I’ve been playing catch up and keep up at the same time; reading accepted, past proposals from other universities, reading articles on the state of evaluation of the CTSA program as a whole, reading articles on different evaluation research from individual CTSA awardees, and reading the many different components of our own proposal in their many different iterations. 

And I’ve been writing; writing a section on the measurement and evaluation program for our Center as a whole, writing annotated versions of the evaluation pieces for the individual components, writing tables and charts, and writing my list of all the people that I’ll send Christmas presents and/or cards to after the middle of January. What I’ll be doing after the grant goes in (and, fingers crossed, gets awarded) is working on all of the evaluation pieces and projects that I’ve spent these weeks describing. I’ll also be working actively with other CTSA award sites, in particular their evaluation teams, on collaborative research projects that will help us determine the effectiveness of the program on a national level. It’s in this latter part that I’ll get to maintain a number of librarian connections, as well as build some new networks of colleagues (and, if evaluators are like librarians, friends).

Do I like it? That’s the other question I get of late. Honestly, I don’t know yet. I like the subject of evaluation and measuring the impact of research. I like the bits that I was able to dabble in while working in the Library. I like research. I like reading and learning new things. I like seeing UMMS from a different perspective. I like the people. I even like the change of pace, even in its whirlwind form. I’m glad that I followed-up on an opportunity and that it’s come to be what it is right now, but it’s still awfully early in the change; too early to give a definitive “thumbs up” or “thumbs down.” 

My highlighted, coffee-stained notebook from that year of hard study. Despite the years, I've never been able to throw it out.

My highlighted, coffee-stained notebook from that year of hard study. Despite the years, I’ve never been able to throw it out.

However, what this early experience reminds me most of is being in graduate school. In particular, it reminds me of the amount of time and effort and work that was involved in learning all of the systemic and cellular physiological mechanisms of exercise for the two courses I took on that subject. I can clearly remember one very bright, sunny, Saturday afternoon in late fall, sitting at the conference table in the room on the 2nd floor of the health sciences building (outside the grad students’ office) with my good friend and classmate, Suzanne Connolly (say it with an Irish accent), working our way through every little step and every confounded enzyme and every change in positive or negative ions to open this or that Calcium channel … all to make one heart beat happen. I can remember talking it through over and over, drawing pictures on the white board, trying to get it, to understand the process, to put all of the pieces together in my head until finally …  BLING!! … the lightbulb went on. And I remember feeling REALLY satisfied. Because it was hard. It was hard, but I’d stuck with it until I got it. And I think that’s why this study day memory stands out so clearly in my memory bank.

I’ve felt that way at other times, too; learning a particular riff on the mandolin or learning to cross-country ski. It takes focus and effort. And it’s about learning, not necessarily about competence. I was (am) a very competent librarian. There was certainly a period of time, early in my career, when I had to put forth a good bit of focus and effort to learn something new, but whenever you’ve been doing something for a good while, the amount of effort you have to expend towards the work decreases. You may still spend the same amount of time and you may still have the same amount (or even more) tasks to attend to, but the amount of effort is different. You’re efficient at what you do. You may not be completely on auto-pilot, but you can probably get to and from work without always remembering the drive.

My first couple of weeks (less a 2-day Christmas break) have been about learning. I’m a long way from being a competent evaluator. The learning curve is steep and challenging, but when I feel overwhelmed, I remember that study day and I remember piano recitals and I remember cruising along on the successes that came from the hard work of learning to be a good librarian. I remember the satisfaction that came with those experiences and I trust that in time, I’ll enjoy the same in my new role. 

Between here and there, I’ll keep sharing the journey. After all, a librarian by any other name is … still a librarian.