Archive | Tools RSS feed for this section

Alternative Metrics ARE Common Metrics

3 Mar

I few weeks back, I was invited by the good folks at Altmetric to take part in a webinar to discuss my use of alternative metrics in my work as an evaluator for the UMass Center for Clinical & Translational Science. The webinar is available online, but for those who might want to see my slides and read the transcript from my part, here you go:

Slide01

Slide02I thought that I’d start with an overview of what’s happening regarding evaluation from the National perspective, since NCATS, the National Center for Advancing Translational Sciences, as the overseer of the CTSA program, steers the ship, so to speak. For those unaware, NCATS is the Center within NIH that oversees the 62 CTSA programs across the country.

The Common Metrics Initiative is a fairly new – or I should say re-newed/re-tooled – working group coordinated by some of the Principal Investigators from the CTSA sites. FYI, the proper jargon for a site is now, hub. So when you see the word “hub,” it refers to an individual CTSA site, such as the UMass Center for Clinical & Translational Science, where I work. Consortium refers to all of the sites, as a whole.

The Common Metrics Initiative came about in an effort to better measure, evaluate, and disseminate the impact of translational science and the concepts behind it. If you think about it, the idea of translational science is that by eliminating some of the barriers and obstacles that exist between biomedical research, clinical research, and clinical practice, discoveries that improve health will move from the lab bench to the bedside, i.e. patient care, faster. The questions of how we measure the truth of this idea is what’s behind establishing a set of common metrics – a uniform, standard set of metrics that measure the speed, efficiency, and quality of this large practice called translational science.

With multiple centers, an infinite amount of programs and research projects, countless individuals involved as researchers, clinicians, students, subjects… you can easily imagine how difficult it is to come up with a common set of metrics that everyone will collect and analyze. But it’s certainly an important thing to do, not only so that we can evaluate our respective individual hubs, but also so that we can compare across hubs.

Briefly, there are four key areas that have been identified as targets for the implementation of common metrics – workforce development (this involves training opportunities for individuals to learn, among other things, how to conduct clinical research); resources and services of each CTSA site; the collective impact of all the programs, functions, and such of an individual site; and the larger CTSA consortium as a whole.

Slide03For today’s talk, I’m going to focus on the area where alternative metrics are most useful. NCATS defines the different resources and services each hub offers as the following:

  • Biomedical informatics
  • Biostatistics, Epidemiology, Research Design and Ethics (commonly known as “the BERD”)
  • Pilot project funding
  • Regulatory Knowledge and Support
  • Community Engagement in Research
  • And Navigational Support – how well those administering hubs connect people to the resources and services that they need.

Slide04Further, I want to focus on the first three bits within this area, BMI, the BERD, and Pilot Funding.

Slide05As the evaluator for the UMCCTS, my job is basically all about answering questions. It’s a good thing that I was a librarian already, since answering questions is the librarians forte. I can also say that one of the things that I love most about being a librarian is answering interesting questions, and my role now certainly offers up a few interesting questions like these:

How effective are our resources and services – from bioinformatics to the parts of the BERD – in contributing to translational research?

When we give people money for pilot research, how well does this research then generate funding for further research? And then, what’s the impact of that research? How is it transforming health care practice and, ultimately, health?

And then the big elephant in the room, not mentioned on this slide, how do we go about answering these questions? The idea of identifying and analyzing a core set of common metrics is one attempt, but what should those metrics be?

These are big, difficult, and very interesting questions.

Slide06Of course, we start with the usual suspects. We count things. How many new projects are initiated? How many people are involved? How many trans-disciplinary collaborations are formed? How many students and new investigators are mentored and trained? How many publications result from the research done? How much new grant funding is obtained to further the work? Remember, the pilot funds offered by CTSA hubs are seed funds. They are meant to help get projects started, not fund them forever.

But what else besides these common metrics can we look to to draw a bigger picture of the success of our work? This, you guessed it, is where we look at alternative or altmetrics.

Slide07So let’s take an example. Here’s a paper authored by one of our researchers and funded, in part, through the resources and services of the UMCCTS. When we’re counting publications as a measure of success, it’s one that I can count. The other thing that I can count that’s fairly traditional is the number of citing articles. We know that this is a relatively good marker for impact – someone citing your work means that they used your work, in some way, to further their own. So the original work is having an effect. In this case, I could point out that 141 other publications needed to use this publication somehow. So we’ve got a reach, in the simplest terms, of 141 – 141 people, projects, research studies, something. This we can say from these two metrics: 1 paper, 141 citations.

But as we all know, today’s communication tools allow for much broader – and easier – dissemination of science. One of my goals, in my work, (you could see it as a challenge and/or an opportunity, too), is to help researchers and funders and other stakeholders appreciate the value of these other tools. To help them see how these tools give us a whole set of other metrics that can help us evaluate the impact of the work.

Slide08This particular paper is always a good example because you can clearly see, via the Altmetric tool, how far it’s traveled beyond the strict confines of scholarly, scientific publications. It’s also reached news outlets and social media users. It’s reached a wide cross-section of people – the general public, patients, other health care practitioners, other researchers in different disciplines. These are also important. We can argue over the level of importance, as compared to citations, but it’s difficult to ignore them – to claim that they have nothing to say when it comes to the measurable impact of this one paper.

The other reason that I like to use this particular paper as an example, besides its impressive Altmetric donut, is because the final author listed – one of the co-PIs for this work – also happens to be the PI for our CTSA site. She’s my boss. The big boss. She’s one of the PIs involved in the evaluation initiatives for NCATs. The first time that I demoed the Donut for her, she loved it. How could she not? Apart from the non-biased reaction that it’s good to see one’s work being shared, it’s also a great ego boost. Researchers, in case you don’t know, are a little bit competitive by nature. They like to see a good score, a good result, a big donut… you name it. They like it.

For those of us trying to reach the goal of bringing altmetrics into favorable light within very traditional disciplines, being able to show this type of example to your stakeholder, in this instance, my boss … it works.

Slide09So day to day, I spend a lot of time at my rock pile doing these sorts of things. I establish collections of publications, related to different groups within the UMCCTS. I maintain those collections regularly – using Collections within MyNCBI in Pubmed, or Scopus and SciVal – two tools available to me thanks to the Library of UMass Medical School. I collect data related to the common metrics outlined by NCATS, but I also collect the altmetrics. I track them all. And then I report on them all via progress reports and infographics (my latest love). It’s an ongoing – never ending – project, but it’s certainly interesting to step back from time to time and look at the big picture, the story, that all of these metrics, together, tell us.

I recently finished the final progress report for the first 5-year funding cycle of our CTSA. It really was impressive to see where we are today, in comparison to where we were just 8 or 9 years ago, when the idea of establishing a clinical and translational science center at UMass Med first took hold.

Slide10“Telling the story” is what my PI reminds me is my job. Using common and alternative metrics, I can tell the story of this one clinical research scholar who, over the past several years, has published 18 papers related to work she’s done in this program. These papers have been cited, she’s worked with many colleagues as co-authors, she’s developed a number of effective collaborations, she’s presented her work locally, regionally, and nationally, and she’s received several new grants to help her continue in her area of research. She’s also reached the public, patients, and other health care providers through multiple means.

Based on all of these metrics, I can write a pretty good story of how well this one doctor is utilizing the resources of the UMCCTS to inform practice and improve care. In a nutshell, I can tell a story of the impact of her research. If I repeat the same for each of our clinical scholars, or a group of researchers utilizing one of our research cores, or one or more of our pilot-funded projects … the story, the picture, gets bigger and, hopefully, clearer. Our Center is making a difference. That’s what we want to show. And that’s possible through the use of all of these tools and metrics.

Slide11Finally, I want and need to give a shout-out to my former colleagues and friends over at the Lamar Soutter Library here at UMass Med. I worked in the Library for 10 years before moving to the UMCCTS a little over a year ago now. It’s the work that I did in the library that first enabled me to build a relationship with our Center, and then inspired me to approach them to do the evaluation work that I do for them now. Kudos to the LSL for all of the initiatives carried out related to scholarly communications and research impact. I think together we’re helping change the environment around here and raising the level of awareness and acceptance of altmetrics.Slide12

Alert! Overstuffed Stocking

17 Dec

HolidayTwinkles

I have a folder on my favorites bar called, “Weekly Blog Post Items” into which I toss bookmarks for sites that I find interesting, helpful, or just fun – things that I think readers of my blog might enjoy, too. As I tossed yet one more URL in it this morning, I realized that it was overflowing. Ah, but ’tis the season for giving and so I give to you, my readers, the gift of a whole bunch of stuff to peruse. Let’s unpack the stocking, shall we?

 

 14 Data Visualization Tools to Tell Better Stories with Numbers is a really nice article by marketing executive, Rob Peterson, posted on the website {grow}. Mapping, timelines, infographics and more are covered, along with links to popular tools to create each. One that I particularly liked is Timeline JS3. I’ve played with it a little bit and it’s pretty cool.

I may have mentioned Print Friendly before, but I love it so much, it’s certainly worth shouting out again. You can either use the website or download the browser button to quickly and easily print only what you want from a website. Don’t want to waste space and ink on that picture? ZAP! It’s gone. Tired of obnoxious ads on your printouts? POW! They’re gone. It’s awesome!

Narrative Visualization: Telling Stories with Data is a very informative and interesting journal article by Edward Segel and Jeffrey Heer of Stanford. It gives insight into balancing visualizations within and/or alongside the narrative.

Every now and then, I go on a hunt to find a presentation platform other than PowerPoint. I admit that while I’ve tried others, I ultimately return to the ruler of the class, but it doesn’t keep me from looking. Beyond PowerPoint: 11 Other Presentation Tools for Small Businesses is a nice collection. Some are likely very familiar to you (Prezi, Haiku Deck, Keynote), but I didn’t know about Projeqt, VideoSlide, or Zentation before I came across this article.

Have you heard of the Relative Citation Ratio (RCR) yet? I hadn’t before reading this article from the American Society of Cell Biology. Those who work in scholarly communications and/or in areas involving tracking and measuring research impact will find it particularly interesting.

LifeCharge describes itself as a “simple elegant journal of your ups and downs.” It’s an app to help you track the things that make you happy, as well as those that bring you down, the goal being that as you look back on the data and the patterns shown, you can make positive changes in your life that might lead you to a happier place. It’s on my “give it a try” list.

Another app on that same list is The Brainstormer. A pocket spinning wheel of nouns, adjectives, and adverbs, that, once the spinning stops, line up to reveal a creative prompt. Here’s one I got… Pursuit > Disco > File Room. Now who couldn’t make a great story, poem, song, drawing, or the like out of that? Ready, Set, GO!

STAT is a new news site, focused on stories at the intersection of science, health, medicine, politics, and money. It’s got big backers from all of these fields behind it, so it may make it in the crowded world of competitors. So far, I’ve found it a very good place to find in-depth reporting on timely, relevant subjects.

This is what the inside of a British cat pub looks like. From Mashable. Need I say anything else?

Back in November, there was a FANTASTIC Google Doodle for Hedy Lamarr. What an amazing woman. Yes, she was a Hollywood star, but did you also know that she’s been inducted into the Inventor’s Hall of Fame? As I said… amazing. Here’s a story about her from the Washington Post.

As it’s that time of year, Slate Magazine published a good list of Best Movies of 2015; Paste Magazine’s 50 Best Songs and 50 Best Albums of the year have a number of nice picks; and you can find a list of all of Mark Zuckerberg’s “Year of Books” choices here. I hope to compile a few of my own “Best Of” lists in the next week or two. When done, I’ll of course share them here.

I discovered ZeeMaps thanks to a tweet by Melissa Rethlefsen. After I posted my “See the World as a Medical Librarian” map a few weeks back, she made her own map using this tool. I immediately bookmarked it.

Less for content than for the step-by-step guide to using Facebook’s Audience Insight tool for seeing trends, themes, etc. among Facebook users, I enjoyed Rob Leathern’s post on Medium entitled, Who is the Average Donald Trump Supporter?

And finally, because celebrating Christmas means, to me, repeating a number of time-worn rituals (I still get up at 5 AM on Christmas morning and run to the tree to see what Santa brought), I share Amy Dickinson’s story of Peanut Jesus, as she titles it, “My modern nativity story: Making Peanut Jesus.” I read it every holiday season. It’s hilarious and touching. I just love it.

P.S. I also read the chapter about the Christmas Pageant from John Irving’s, “A Prayer for Owen Meany.” Owen as Baby Jesus rising up out of that crib in the manger… I laugh until I cry.

Happy Holidays to you all! Thank you for following along with me another year.

The Art of Collaboration

12 Nov

[The following is my monthly column for the November issue of the UMCCTS newsletter.]

One of the goals of the UMCCTS is to promote and facilitate collaboration across departments and disciplines, thus effectively reducing barriers between the basic and clinical sciences, and ultimately speeding the pathway between the discovery and implementation of new treatments, therapies, and the like that improve health. One means of demonstrating collaboration is through co-authorship. The networks that develop between authors of publications give us a picture of how individuals are connected and where collaborations exist.

Social network analysis is the process of investigating social structures through the use of network and graph theories. It characterizes networked structures in terms of nodes (individual actors, people, or things within the network) and the ties or edges (relationships or interactions) that connect them. (Wikipedia, Social Network Analysis

For this month’s column, let’s look at an example of a social network analysis that shows the co-authorship relationships between members of the Division of Health Informatics and Implementation Science in the Department of Quantitative Health Sciences (QHS). QHS is one of the newest departments at UMMS, with several of the senior faculty arriving on campus only about 6 years ago. The research that the Department does in developing innovative methodologies, epidemiological research, outcomes measurement science, and biostatics is integral to the nature of clinical translational research. By examining the co-authorship relationships of members of the Health Informatics group, we get a snapshot of how well these faculty members are connecting with other departments, other disciplines, and even other institutions. In short, we see how and where collaborations have developed and thus how well the UMCCTS goal of building them is being met.

To do this analysis, we first need to identify all of the publications authored by at least one of the Division’s faculty members for the period of time that s/he has been part of the Division, as well as all of the unique co-authors associated with these papers. In doing this, I found 221 publications authored by 716 different individuals. Using Sci2, a toolset developed at Indiana University, I was able to analyze the patterns and create a visualization showing the connections between the co-authors.

Informatics Division CoAuthor Network

One thing that we clearly see is that several faculty members are prominent hubs in the network, meaning they co-author many papers with many people. Drs. Houston and Allison are the most obvious examples here. We can also see that a number of branches grow from the periphery. At the base of each of these is a faculty member from the Division (counterclockwise from upper right, Drs. Cutrona, Hogan, Shimada, Mattocks, and Yu). Finally, we note that even hubs that are less connected to the clustered middle, e.g. Drs. Yu and Pelletier, are still linked, representing the reach of the collaborative network that the Division has formed over the past years.

Tools like Sci2, Scopus, SciVal, and ISI Web of Science provide another way, i.e. a visual demonstration, of the success of our programs and the impact of the translational science being done by the members of the UMCCTS.

Sci2 Team. (2009). Science of Science (Sci2) Tool. Indiana University and SciTech Strategies, https://sci2.cns.iu.edu.

Share and Share Alike

1 Oct

One of my favorite books from the past few years is Austin Kleon’s, Steal Like an ArtistI’ve mentioned it in several previous posts (search “Austin Kleon” on the site and you’ll find them), mostly because I continue to pop back to it on a regular basis. It’s filled with plain, simple, good thoughts to inspire your creative side. I also follow Austin on Twitter. Awhile back, he declared that he was going to shift from immediately tweeting out lots of ideas, project updates, and interesting things he came across online to putting them all in an indexed version that he’d send out via his Tumblr account on Fridays. Of course, as soon as I saw this announcement I signed up for his email list and ever since, his Friday email to me has become something that I look forward to.

My new role as an evaluator finds me doing a lot of things that I’m hard pressed to chronicle as I once did for my work in the library world. In part, I think it’s because I spend a great deal of time learning new things and/or putting newly learned skills into action. It takes time and energy that ultimately takes away from my abilities to come up with interesting musings for this blog. That said, I’m not about to give up my blogging habit. It means too much to me. After lots of thinking about how to revitalize it, the thought came to me to take Austin’s advice and steal an idea … from him!

Thus, I’ve decided to shift the pattern of own blog a bit – at least for awhile – and turn it into a way to share with you, my readers and followers, some of the cool and interesting and inspiring and, dare I hope, helpful things that I come across weekly in my work and play. So here we go … here are a few things from the past several weeks (I’m cheating already, but it’s the start of a new thing and thus allowed). Enjoy!

  1. It only seems fair that I give a tip of the hat to Mr. Kleon to start. Besides his books, I also enjoyed watching the video from a terrific talk that he gave to an audience at Google a few years ago. It’s a wonderful summary of his theory on stealing and some inspiring words to anyone seeking to get out of the way of themselves when it comes to creativity.
  2. Juice Analytics is a data analysis and design firm in Atlanta that provides visualization services to businesses and organizations. They also freely offer a number of great resources for learning these skills, including white papers, video tutorials, and the book, Data Fluency (not free, but well worth the $21.59 price tag for my Kindle version). One of the best resources on their freebie page is “30 Days to Data Storytelling,” a guide to … well, it’s pretty self-explanatory, isn’t it? It’s a list of videos, tutorials, articles, etc., a few a day for 30 days, to help you understand how to use data to tell your story. Good stuff.
  3. Back at the end of the summer, just as school was ready to gear up, Slate published a series of blog posts during one week under the banner, What Classes Should I Take? The list is fascinating and the posts very well written. Two that I liked in particular were, The Secret Technique for Learning How to Code: Step 1. Don’t Be Intimidated, by Victoria Fine, and What are the Odds: To Learn to Think Critically, Take a Statistics Class, by Laura Miller. These two are most relevant to anyone in the library, information, or evaluation worlds. I also found the advice to take Art History, Public Speaking, and No Class at All, quite valuable. The entire series was great.
  4. The Noun Project – Icons for Everything – is pure awesomeness. A gazillion free icons to drop and drag and plop into place OR inspire you to make your own.
  5. One thing that I do often in my job is doodle pictures to tell the story of a particular group of researchers or a research center. Fancy word, infographics. Since I started sharing some of these on this blog and other places, several colleagues and friends have asked for advice on tools to use to make them. I tend to draw my own in Illustrator and/or Powerpoint, but there’s a handy list of 10 Free Tools for Creating Infographics on the Creative Blog website.

Finally, I think I’d like to add one consistent thing for each of these lists/posts. I’m going to call it, What’s On My Desk Right Now. Right now, it’s this:

Visual Storytelling

Visual Storytelling: Inspiring a New Visual Language, edited by Klanten, Ehmann, & Schulze, and available through Gestalten. I learned about this book after stumbling upon an interview with Jonathan Corum, the graphics editor for science at the New York Times. He’s one of many featured in this book and I can’t wait to dive into it. Now. Lunchtime reading!

Illustrated Podcasting

2 Sep

My podcasting workshop last night wasn’t quite all that it was advertised to be, but I definitely learned a lot and feel pretty prepared to tackle the task. I was hoping to leave the class with a finished and distributed episode (per the course description). I like classes and workshops that promise such. It wasn’t to be, but for the $17.50 fee, plus the bonus of catching up with my friend, fellow librarian, blogger, beer connoisseur, and baseball lover, Dan, before, it was more than worth it. I sketched my notes, per usual, and share them here. And soon, catch the podcast I’m going to create!

PodcastClass_1

PodcastClass_2

PodcastClass_3

PodcastClass_4

For those keeping track of my office supplies, these sketchnotes are drawn in a FieldNotes brand ruled memo book, carried in my “never go anywhere without it” handmade “Everyday Carry” cover, with a refillable Pentel EngerGel pen that I’ve managed to hang on to for several years now. I hope that I never lose it, as both Rosanne Cash and Amy Dickinson have used it to pen a few words to me. It’s a treasure.

January 9th – ALREADY?!

9 Jan

It’s a good thing that I didn’t make any New Year’s resolutions related to my personal writing, because I’d have to report a failure already. That said, the CTSA grant proposal that everyone has been working ’round the clock on for weeks now is very close … oh so very close … to being put to bed, which in this case means submitted. And then I’ll be able to start focusing on how to approach doing the new job that I’ve been hired to do. Up until now, I’ve only been writing what I’ll do. Next stop, figure out how to do what I said I’d do. I’ve already joined the American Evaluation Association and signed up for one of their upcoming coffee break webcasts.

An aside… I think the idea of coffee break webcasts – 30-minute weekly sessions that focus on a particular topic, led by different members of the organization – is a TERRIFIC idea. I know that I belong to a few organizations that are struggling to define and/or create the real benefits of membership and such a simple thing as a regular, free, short-and-sweet-yet-interesting webcast is just that sort of thing.

For today, I at least wanted to send up a post with a few fun things I’ve come across over the past couple days/weeks – some delayed candy canes, if you will:

  • The Spudd – it’s The Onion of medical and pharma news. Hilarious. I discovered it just this very morning, thanks to a hilarious post shared on Twitter by my friend, Dean Hendrix. 
  • How Reddit Created the World’s Largest Dialogue between Scientists and the General Public is a very good blog post by Simon Owens. I’m fascinated with scientific communication and, in particular, efforts to bring the scientific community together with the general public. We are a scientifically illiterate culture at our own peril. I love what’s happening on this online community and so I’ve set up a Reddit account and plan to follow along for awhile. 
  • Finally, for anyone curious about public health and/or epidemiology and NOT interested in returning to school ever again <hand raised>, I came across an on-demand course from the University of North Carolina at Chapel Hill. I confess that I’ve signed-up and failed at several MOOC’s, mostly because of timing. I’m really happy to find a relevant, on-demand one and hope to work through it soon. I have a feeling that doing a course on my own, at my own pace, and at my own convenience will work well for me, especially now as I juggle all of the new tasks of a new job.

Back to the grindstone here. Happy New Year to all of my readers and followers! You make blogging fun.

My 3 New Year's Resolutions for 2015. No progress yet!

My 3 New Year’s Resolutions for 2015. No progress yet!

FlipQuiz: A Great New Teaching Tool

24 Sep

I’m teaching Health Sciences Librarianship for the University of Rhode Island’s Library and Information Studies graduate program this semester. Sometimes, I think we can learn as much being a teacher as we do being a student. In this case, I’ve been learning to use a number of new tools, new concepts, and new ways to teach online, and for distance learners. I’ve had to read a lot of theory and try out a bunch of resources and it’s still only September!  Fortunately, I think my students are both patient and open to the trial and error of my learning. 

This week, I discovered a terrific new tool that I want to share via this blog. I think others will find it pretty useful, too. If you’re a fan of learning via games (as I am), give FlipQuiz a try!

NCBI Game

Feel free to click on the game board and play, but don’t share answers with my students! 🙂