Alternative Metrics ARE Common Metrics

3 Mar

I few weeks back, I was invited by the good folks at Altmetric to take part in a webinar to discuss my use of alternative metrics in my work as an evaluator for the UMass Center for Clinical & Translational Science. The webinar is available online, but for those who might want to see my slides and read the transcript from my part, here you go:

Slide01

Slide02I thought that I’d start with an overview of what’s happening regarding evaluation from the National perspective, since NCATS, the National Center for Advancing Translational Sciences, as the overseer of the CTSA program, steers the ship, so to speak. For those unaware, NCATS is the Center within NIH that oversees the 62 CTSA programs across the country.

The Common Metrics Initiative is a fairly new – or I should say re-newed/re-tooled – working group coordinated by some of the Principal Investigators from the CTSA sites. FYI, the proper jargon for a site is now, hub. So when you see the word “hub,” it refers to an individual CTSA site, such as the UMass Center for Clinical & Translational Science, where I work. Consortium refers to all of the sites, as a whole.

The Common Metrics Initiative came about in an effort to better measure, evaluate, and disseminate the impact of translational science and the concepts behind it. If you think about it, the idea of translational science is that by eliminating some of the barriers and obstacles that exist between biomedical research, clinical research, and clinical practice, discoveries that improve health will move from the lab bench to the bedside, i.e. patient care, faster. The questions of how we measure the truth of this idea is what’s behind establishing a set of common metrics – a uniform, standard set of metrics that measure the speed, efficiency, and quality of this large practice called translational science.

With multiple centers, an infinite amount of programs and research projects, countless individuals involved as researchers, clinicians, students, subjects… you can easily imagine how difficult it is to come up with a common set of metrics that everyone will collect and analyze. But it’s certainly an important thing to do, not only so that we can evaluate our respective individual hubs, but also so that we can compare across hubs.

Briefly, there are four key areas that have been identified as targets for the implementation of common metrics – workforce development (this involves training opportunities for individuals to learn, among other things, how to conduct clinical research); resources and services of each CTSA site; the collective impact of all the programs, functions, and such of an individual site; and the larger CTSA consortium as a whole.

Slide03For today’s talk, I’m going to focus on the area where alternative metrics are most useful. NCATS defines the different resources and services each hub offers as the following:

  • Biomedical informatics
  • Biostatistics, Epidemiology, Research Design and Ethics (commonly known as “the BERD”)
  • Pilot project funding
  • Regulatory Knowledge and Support
  • Community Engagement in Research
  • And Navigational Support – how well those administering hubs connect people to the resources and services that they need.

Slide04Further, I want to focus on the first three bits within this area, BMI, the BERD, and Pilot Funding.

Slide05As the evaluator for the UMCCTS, my job is basically all about answering questions. It’s a good thing that I was a librarian already, since answering questions is the librarians forte. I can also say that one of the things that I love most about being a librarian is answering interesting questions, and my role now certainly offers up a few interesting questions like these:

How effective are our resources and services – from bioinformatics to the parts of the BERD – in contributing to translational research?

When we give people money for pilot research, how well does this research then generate funding for further research? And then, what’s the impact of that research? How is it transforming health care practice and, ultimately, health?

And then the big elephant in the room, not mentioned on this slide, how do we go about answering these questions? The idea of identifying and analyzing a core set of common metrics is one attempt, but what should those metrics be?

These are big, difficult, and very interesting questions.

Slide06Of course, we start with the usual suspects. We count things. How many new projects are initiated? How many people are involved? How many trans-disciplinary collaborations are formed? How many students and new investigators are mentored and trained? How many publications result from the research done? How much new grant funding is obtained to further the work? Remember, the pilot funds offered by CTSA hubs are seed funds. They are meant to help get projects started, not fund them forever.

But what else besides these common metrics can we look to to draw a bigger picture of the success of our work? This, you guessed it, is where we look at alternative or altmetrics.

Slide07So let’s take an example. Here’s a paper authored by one of our researchers and funded, in part, through the resources and services of the UMCCTS. When we’re counting publications as a measure of success, it’s one that I can count. The other thing that I can count that’s fairly traditional is the number of citing articles. We know that this is a relatively good marker for impact – someone citing your work means that they used your work, in some way, to further their own. So the original work is having an effect. In this case, I could point out that 141 other publications needed to use this publication somehow. So we’ve got a reach, in the simplest terms, of 141 – 141 people, projects, research studies, something. This we can say from these two metrics: 1 paper, 141 citations.

But as we all know, today’s communication tools allow for much broader – and easier – dissemination of science. One of my goals, in my work, (you could see it as a challenge and/or an opportunity, too), is to help researchers and funders and other stakeholders appreciate the value of these other tools. To help them see how these tools give us a whole set of other metrics that can help us evaluate the impact of the work.

Slide08This particular paper is always a good example because you can clearly see, via the Altmetric tool, how far it’s traveled beyond the strict confines of scholarly, scientific publications. It’s also reached news outlets and social media users. It’s reached a wide cross-section of people – the general public, patients, other health care practitioners, other researchers in different disciplines. These are also important. We can argue over the level of importance, as compared to citations, but it’s difficult to ignore them – to claim that they have nothing to say when it comes to the measurable impact of this one paper.

The other reason that I like to use this particular paper as an example, besides its impressive Altmetric donut, is because the final author listed – one of the co-PIs for this work – also happens to be the PI for our CTSA site. She’s my boss. The big boss. She’s one of the PIs involved in the evaluation initiatives for NCATs. The first time that I demoed the Donut for her, she loved it. How could she not? Apart from the non-biased reaction that it’s good to see one’s work being shared, it’s also a great ego boost. Researchers, in case you don’t know, are a little bit competitive by nature. They like to see a good score, a good result, a big donut… you name it. They like it.

For those of us trying to reach the goal of bringing altmetrics into favorable light within very traditional disciplines, being able to show this type of example to your stakeholder, in this instance, my boss … it works.

Slide09So day to day, I spend a lot of time at my rock pile doing these sorts of things. I establish collections of publications, related to different groups within the UMCCTS. I maintain those collections regularly – using Collections within MyNCBI in Pubmed, or Scopus and SciVal – two tools available to me thanks to the Library of UMass Medical School. I collect data related to the common metrics outlined by NCATS, but I also collect the altmetrics. I track them all. And then I report on them all via progress reports and infographics (my latest love). It’s an ongoing – never ending – project, but it’s certainly interesting to step back from time to time and look at the big picture, the story, that all of these metrics, together, tell us.

I recently finished the final progress report for the first 5-year funding cycle of our CTSA. It really was impressive to see where we are today, in comparison to where we were just 8 or 9 years ago, when the idea of establishing a clinical and translational science center at UMass Med first took hold.

Slide10“Telling the story” is what my PI reminds me is my job. Using common and alternative metrics, I can tell the story of this one clinical research scholar who, over the past several years, has published 18 papers related to work she’s done in this program. These papers have been cited, she’s worked with many colleagues as co-authors, she’s developed a number of effective collaborations, she’s presented her work locally, regionally, and nationally, and she’s received several new grants to help her continue in her area of research. She’s also reached the public, patients, and other health care providers through multiple means.

Based on all of these metrics, I can write a pretty good story of how well this one doctor is utilizing the resources of the UMCCTS to inform practice and improve care. In a nutshell, I can tell a story of the impact of her research. If I repeat the same for each of our clinical scholars, or a group of researchers utilizing one of our research cores, or one or more of our pilot-funded projects … the story, the picture, gets bigger and, hopefully, clearer. Our Center is making a difference. That’s what we want to show. And that’s possible through the use of all of these tools and metrics.

Slide11Finally, I want and need to give a shout-out to my former colleagues and friends over at the Lamar Soutter Library here at UMass Med. I worked in the Library for 10 years before moving to the UMCCTS a little over a year ago now. It’s the work that I did in the library that first enabled me to build a relationship with our Center, and then inspired me to approach them to do the evaluation work that I do for them now. Kudos to the LSL for all of the initiatives carried out related to scholarly communications and research impact. I think together we’re helping change the environment around here and raising the level of awareness and acceptance of altmetrics.Slide12

Occupy Leap Day

29 Feb

Here’s a post that I wrote for my personal blog 4 years ago – the last time we celebrated a Leap Day. I’m not able to occupy my Leap Day in such a way this year, but I hope some of you are. Here’s to doing something creative on your extra day!

Happy Leap Year!

 

blahg, blahg, blahg...

Every four years we’re given a present, an extra 24 hours that we bundle up into one day that we call “Leap Day”. Where does this time come from? Well, astronomically speaking, it takes the earth approximately 365 days and 6 hours to travel around the sun. Add up those 6 hours for 4 years and you’ve got 24 (thank you Ms. Summers for all of those multiplication flashcard drills in 4th grade). To keep our old calendars in line, we add that extra day in during that 4th year and call it “Leap Year”.

It dawned on me this morning that if there is one day that should really be given to people as a bonus day, a day to do whatever you wish, a holiday if you will, it is Leap Day. It only seems fitting as (1) we get by just fine without it 3 out…

View original post 237 more words

Crossing the Radar Screen

5 Feb

radar-333574_960_720It’s Friday and it’s snowing here in Worcester – all of the makings of a quiet afternoon. I’ve spent the day mostly working through a book that I recently bought, Tableau Your Data! It’s a lot to take in, so I thought I’d take a break and clear out my “Weekly Blog Post Items” bookmark folder. Here are some fun and interesting finds that crossed my radar screen during the past week:

Determined to hone my data visualization chops, I’ve been on the lookout for interesting sources of data to use for practice. The U.S. Census Department’s website is a great spot, of course, but a special gem that I found hidden on it is Stats for Stories. Here, you’ll find statistics related to stories that are in the news, calendar events and/or holidays, and more.  

It’s 2+ hours long so I’ve hardly sat and watched the entire thing yet, but what I’ve seen of the keynote address by Christian Chabot and Chris Stolte on the “Art of Analytics” at Tableau Conference 2014 is quite fascinating. Data visualization as an art form – it’s a topic that draws me in.

Obsessive fans (who me?) of the TV show, Law & Order, along with its many iterations will find Cecilia Esther Rabess’ latest entry in her McSweeney’s column, Mostly Uninformative Infographics, hilarious and oh, so true. … About Law & Order: Special Victim’s Unit

My wife has been co-teaching Sunday school these past few months using a curriculum called, “D’oh, God!” It’s based around episodes of The Simpsons. Naturally, when I learned about Frinkiac, the database of 3 million+ screen captures from the show, I had to pass it along. Enjoy!

The Washington Post’s story, “What Ivy League Students are Reading That You Aren’t,” along with the data source for it, the Open Syllabus Explorer database, both fascinated me this week. 

If you’re curious about the source of words and phrases in the English language, you’ll likely find Arika Okrent’s YouTube channel awesome. Okrent is a contributor to the magazine, Mental Floss. I’ve subscribed to it for years, preparing myself for that “Jeopardy!” tryout that I just know I’m going to be invited to some day.

Virginia Woolf made famous the idea of “a room of one’s own.” I’m so very fortunate to have a studio space in an old factory mill in town where I can go and be creative in any and every fashion. It’s my space. My room of my own. Bored Panda’s “100 Famous Artists and Their Studios” is a wonderful photo trip through the rooms of some incredibly talented people. I found it inspiring.

Finally, the SuperBowl is this Sunday. I’m likely in the very small minority who tunes into the game to watch the game. I turn to a different channel during the half-time show and I mute all of the commercials. I realize that folks pay a gazillion bucks for these spots, but I always mute (or fast-forward through) commercials. Bleh! That said, these two spots made for Sunday’s game but released earlier got me. Dachshunds and singing sheep. What could be better?!  

and

Happy Friday, everyone!

Show Me the Numbers

3 Feb

I’ve noticed how ever since I became an evaluator, I’m much more in tune to numbers. This isn’t to say that I never paid any attention to numbers before, but now, when I hear stories on the radio or I read articles in my local newspaper, I look more closely at what’s being reported regarding those numbers. What’s really being said? And more, I find myself asking, “What do these numbers really represent?” Here’s an example:

This morning, I was listening to a story on NPR about the voter turnout in this week’s Iowa caucus. Specifically, the story was about the turnout among younger voters (17-29 years of age) in Iowa and what, if anything, this turnout says about this voting bloc nationally.

Aside: You can find interesting data regarding the Iowa electorate (as well as other states) on the U.S. Census Department’s website. You can find specifics regarding the turnout of younger Iowa voters on the website of CIRCLE (The Center for Information and Research on Civic Learning and Engagement).

But back to the NPR story… Renee Montagne interviewed Kei Kawashima-Ginsberg, director of the Center for Information and Research on Civic Learning and Engagement at Tufts University, about these millennial voters. Phrases like, “record numbers” make my ears perk up. “What was the record?” I wonder. “What are we talking about?” In brief, Kawashima-Ginsberg stated, “The youth turnout was 11.2%.”

“11.2% of what?” I ask out loud in my car, to no one.

“On the Republican side, Ted Cruz received 27% of the votes, Mark Rubio 24%, and Donald Trump 19%.”

Again I ask, “27% of what?” No one answers.

Bernie Sanders, I’m told, won 84% of the Democratic vote, compared to Hillary Clinton’s 14%.

“Wow! 84%. That’s a lot! You do keep reporting how he’s winning the hearts of young folks.”

I pull out the note pad that I keep in the dashboard cubbie of my car and write down, “Young voters 84%, 14% // 11% = x” I put the note in my pocket, determined to figure out what these numbers mean. Later, I did.

The total number of young people, defined here as voters between the ages of 17-29, that participated in the Iowa caucus was 53,215. What’s that look like? I need a visual reference. I think of this demographic and I think of college. It’s a natural reference-point for me, a college grad. When I think of college and crowds, I think football. (Plus, the SuperBowl is but a few days away. Think football.) Thus, to give myself the visual that I need, I decide to compare these numbers to the capacities of various college football stadiums. Here’s what I found…

… 53, 215 people equals a sold-out crowd for a football game at Rutgers University’s High Point Solution Stadium.

RUFootballStadium

High Point Solutions Stadium, Rutgers University, East Rutherford, NJ

Okay, that’s a good-sized crowd. Granted, it’s not quite half of the capacity of the University of Michigan’s stadium, but let’s remember, it’s Iowa, a state who’s population makes up .97% of the United States as a whole. Michigan is up there at 3.11%. (All of this data comes from Census.gov.)

Of these 53,215 caucus-goers, 22,415 were Republicans and 30,800 were Democrats. Bernie Sanders won the support of 84% of those 30,800, or approximately 25,800 young people. I need a reference. What do 25,800 people look like? A sold-out crowd at my alma mater, James Madison University’s Bridgeforth Stadium. Go Dukes!

Bridgeforth Stadium

Bridgeforth Stadium, James Madison University, Harrisonburg, VA

Hillary Clinton’s 14%, or 4,312 youthful supporters from Tuesday night, could fit in at Sacred Heart University’s (Fairfield, CT) Campus Field.

Campus Field

Campus Field, Sacred Heart University, Fairfield, CT

Ted Cruz and his 27% of young Republicans (5,828) fill up the Butler Bowl of the Butler University’s Bulldogs in Indianapolis, IN.

Butler Bowl

Butler Bowl, Butler University, Indianapolis, IN

Mark Rubio’s 5,155 (24%) supporters would fill the stands of the University of Rhode Island’s Rams Meade Stadium.

Meade Stadium

University of Rhode Island, Meade Stadium, Kingston, RI

And finally, Donald Trump’s 4,483 supporters, or 19% of the young Republican caucus-goers, would fit nicely in Bryant College’s (Rhode Island) Bulldog Stadium. Or perhaps, more apropos, they could stay approximately 3 to a room in the 1,250 “deluxe guest rooms and palatial suites” of the Trump Taj Mahal casino in Atlantic City.

Bulldog Stadium

Put into these contexts, the numbers make so much more sense to me. Sure, 25,800 people (that 84% Bernie came home with) is a lot of people, but in perspective, my alma mater isn’t exactly a gigantic school. It’s a good-sized school, mind you, but it’s hardly representative of the number of people who might vote in a general election, even if they could all agree on anything, in mass, besides cheering for the Dukes.

Additionally, these stories say an awful lot about how numbers and statistics get used in our reporting. “The American People,” a phrase that every single politician, pollster, and news junkie talking head over-uses means … what A percentage of a percentage of a percentage of a percentage of people is generally a number way smaller than an image that “The American People” conjures up. It’s also, more than likely, a smaller sample size of ideas and beliefs, morals and behaviors, arguments and agreements, and problems and solutions than the 323,000,000 people in the United States hold in total. 

Yes, the political season in America is just getting rolling and it’s a great time to pay attention to the numbers reported, seek out sites for trustworthy statistics, do some math yourself, and hone up on your data fluency skills. (That last bit is a nod to a terrific book, Data Fluency, from the smart folks at Juice Analytics. Check it out.)

 

#OscarSoBlackAndWhite

29 Jan

To hone my data visualization and information graphics chops, I’ve been seeking out and collecting data on topics of interest to me. Here’s one that bugs me around this time every year.

Oscar So Black and White

And the Oscar Goes To: Best Picture

15 Jan

OscarThe Oscar nominations are out. No big surprises. When it comes to patting themselves on the back, the industry that is Hollywood is still overwhelmingly white and male. Still, it was a year for several good films and many wonderful performances and, per usual, I have a lot to catch up on before the awards are handed out next month. Cue up the movie tickets, Netflix, and popcorn.

For fun, I wondered what it would be like to name my personal Oscar nominations and winners this year. Oh, I don’t mean making picks based upon the movies of last year, but the events of my life during that time. What would be the Best Picture of my 2015? Who would be the Best Actress? Best Director? What was the Best Score, the background music of my year?

I sat down with pen and paper and started my lists. It’s more difficult that I thought and thus will take a few posts, but let’s start with what those darned Oscar celebration directors always make us wait until the end to find out – Best Picture. No need to stay up past midnight here.

The 2015 nominees for Best Picture in my year are:

ACC

It’s a long way from the Library to the 7th floor of the ACC.

The Road Less Traveled – A medical research librarian leaves the familiar confines of the library and her library kin to explore the highway of evaluation in clinical and translational  science. Along the way she meets up with intense grant writing, crazy deadlines, people who speak a different language, and much packing and unpacking of office boxes. Will it be a cliffhanger or a “happily ever after”? You decide.

*****

Little Snow

Dogs always steal the snow, er… show.

Snowpocalypse 2015 – The bustling, blue collar, chip-on-its-shoulder city of Worcester, Massachusetts is pummeled with snow the likes of which it cannot remember. Almost 120 inches of snow falls, leaving the City buried in challenges, but full of heart as the citizens all get behind the race to claim the title, “Snowiest City in the US.” No spoilers here. You’ll have to check out the Golden Snow Globe to see who won.

*****

IMG_2052

Austin, Texas. The backdrop alone makes for a winning film.

SwingTime – A bunch of medical librarians land in Austin, Texas and discover that honky tonks and margaritas and Texas Swing are all right up their alley. Meeting? Was there also a meeting? Think of this one as Todd Phillips writes a movie with smart people in the cast.

*****

IMG_2282

It’s ideal, but is it a winner?

Our House in the Middle of the Street – Adopting the title of the hit song by the band, Madness, back in the 1980s, the attempt to buy a home becomes maddeningly complicated at every turn possible. A Mad, Mad, Mad, Mad World it was, but this picture avoids the pitfalls of  The Money Pit and becomes an instant classic. Home, Sweet Home.

*****

Four Friends

Who will fall into the drama?

The Big Thrill – Four friends gather by a lake for a weekend of reminiscing. Twenty five years may have gone by between meetings, but the reunion is filled with laughter and tears. Lawrence Kasdan’s influence knows no bounds.

***************

Wow! What a slate. There’s not a non-deserving picture in the bunch. Hand me the envelope there, judges. And the Oscar for Best Picture of 2015 goes to…

Our House in the Middle of the Street! 

This is the first Oscar win for first-time homeowners Sally and Lynn. The sentimental favorite, yes, but who can argue? We can now sit in the comfort of a warm and cozy home and watch movies forever. Definitely a “Happily Ever After” feature.

Stay tuned for more. We’ll be back after a commercial break.

It’s Still the Giving Season

7 Jan

Yes, yes… the official holiday season is a little bit behind us now, but my ChristmasChristmas tree is still up and I celebrate as long as possible, i.e. until said tree becomes a fire hazard, so in the spirit of giving, here are a few goodies I’ve discovered over the past couple of weeks. Enjoy!

Reading Lists! Oh how I love them and this time of year is always ripe with such. The folks at TED Talks produced a nice one – 58 books recommended by TED speakers – and my favorite independent bookstore, Powell’s (Portland, OR)  offered up a selection of their favorite nonfiction books of 2015.  Here’s a list from Austin Kleon, who I mention here often – his reading list from the past year. And finally, if you missed reading along with Facebook’s chief, Mark Zuckerberg during his A Year of Books group, it’s not too late to see all of the books he read and discussed. BTW, if you’d like to run with Mark this year, you can join his new Facebook group, A Year of Running. 365 days = 365 miles. Go for it!

And while we’re on lists, I did my favorite music list in my last post, but afterward came across NPR’s poll results of their listeners favorite albums of 2015. More for your listening pleasure.

Switching gears… did you make a New Year’s Resolution to learn something new? I did. I’m taking banjo lessons. Hot dog! But I’m taking them in person because I’ve tried in vain to learn via online classes and Youtube videos. That said, there’s plenty of subjects well suited for the virtual world and the post on Medium, The 37 Best Websites to Learn Something New, will point you in the right direction.

Sadly, one thing that always gets me a little down when I think about and/or pursue learning something new is that I’m getting older. Older every day. I fear that I’ve missed out on ever becoming really good at something that I might want to do or be. I’m sure we all might feel that life crisis from time to time. In the midst of such anxiety last week, my daily email from The New Yorker arrived with a link to a great piece by Malcolm Gladwell, Why Do We Equate Genius with Precocity? I liked it a lot and found hope in the thought that maybe I’m just a late bloomer.

Just for fun – “The lyrebirds of Australia were highly mysterious and rarely seen until one fell in love with an elderly widow in 1930.” Surely you will want to listen. How could you not?

 And finally, perhaps one of the coolest things that crossed my radar this week is this wonderful collection of infographics from the folks at Wait But Why. Could you imagine looking at 90 years of living in a more creative way? Great stuff.

That’s all for this week. Here’s to a Happy, Healthy, Peaceful, and Curious Creative-filled 2016 for all!

Follow

Get every new post delivered to your Inbox.

Join 1,953 other followers