Can We Measure Social Media Sophistication? If So, How?
Can we validly and reliably measure social media sophistication? In Part 2 of a two-part essay, I share my perspective on the challenges in trying to create a universal measure, and offer some thoughts on what a tool to measure social media sophistication might include. The thoughts and perspectives of others are welcome.
In 2009 I created a social media quiz, practically on a whim. Over the years I’ve converted the simple quiz into a more complex assessment of social media sophistication, but it’s still very unscientific. It’s great for ice breakers, workshops, and discussions, but its lack of reliability and validity limits its use beyond that.
Over the years, the social scientist in me has contemplated the question of whether there should be a “real” tool to measure social media sophistication. This is Part 2 of a two-part essay that captures some of my current thinking. In Part 1 I provided my perspective on whether understanding social media sophistication is important and whether there’s value in trying to create a universal measure to assess it. Now I’ll address the question of whether we can create a means to measure social media sophistication and share some thoughts on what a measure might include.
Can We Create a Measure of Social Media Sophistication?
The attempt to create a valid and reliable tool to measure social media sophistication would hardly be easy for many reasons, including:
Identifying platforms and tools to include. Social media is in a constant state of flux, which means it’s virtually impossible to identify a definitive list of platforms, tools, and apps around which to build a measure. Finding the right balance between services people use in their personal lives (e.g., Facebook, Flickr) and those they use in their professional lives (e.g, LinkedIn, SlideShare), as well as those that would be used in organizations (e.g., Yammer, SharePoint), adds even more complexity to the issue. And of course there’s also the challenge of creating bias by including and excluding services that are predominantly used in specific countries (e.g., Sina Weibo in China, VK in Russia).
Determining what to measure – and how. What are the most reliable and valid ways to assess social media sophistication? What should we measure? Knowledge of aspects of various platforms, tools, and apps? Frequency of visits? Time spent? Number of connections? Degree of activity? Level of customization? Tests of skill? It is no easy task to determine what will indicate different degrees of sophistication and decide whether and how to standardize those measures across different platforms, tools, and apps.
Using some kind of simple survey or quiz instrument creates the risk of self-report bias, as responses can be skewed by the inaccuracies of people’s perceptions and memories. If you ask someone how much time they spend on a given platform, for example, they may underreport their total time because they don’t want people to think they’re a slacker who has nothing better to do than hang out in cyberspace. They can also claim to have skills and abilities they don’t have, and/or that are significantly weaker than they think they are. Hashtags are a great example of this, as most people are still using them incorrectly.
A skills-based test, in which people are provided with a simulated social media platform and asked to do things like create a profile, upload a photo or video, share a link, comment on someone else’s post, and send a private message reduces the risk of self-report bias but is a much more complex – and costly – approach.
Concerns about privacy. Do people really want organizations to know all the sites on which they have a presence, as well as the extent of their knowledge, skills and usage? Privacy is one of the biggest issues we have to grapple with as digital technology continues to evolve, and people have and will become increasingly hesitant to divulge too much about their digital habits due to concerns about where and how that information may be used.
A related challenge is who should create the measure and own the data? Should this be a strictly academic endeavor? Should the assessment be free, or is it appropriate for it to be developed and sold by a commercial testing company? How should collected data be aggregated and reported?
Cut-off points and labels. How do we determine what distinguishes degrees of social media sophistication? What categories do we create, and how many do we need? Where are the cut-off points? There are no definitive answers to any of those questions, and coming up with them will be complicated to say the least. And no matter how sophisticated the test may be, it’s still a relatively simple measure that can’t fully capture the complexity of someone’s knowledge, understanding, and usage.
Whatever the categories and breakpoints are, there’s a risk that the results of a social media sophistication measure will result in an inappropriate use of labeling and stereotyping. For example, if I were assessed based on my personal Twitter usage (versus my professional usage), I may appear to be an unsophisticated user because I only follow a handful of accounts and never tweet (because I use it as a de facto newspaper). But that inactivity reflects a strategic decision on my part about how to best use the tool given my goals and objectives – in other words, it may in fact reflect a high level of sophistication.
Employment concerns. Attempting to measure social media sophistication in an employment context creates all kinds of risks related to a variety of labor laws.
Thoughts on What a Tool to Measure Social Media Sophistication Might Include
I did a few quick internet searches and reviewed some of the existing quizzes and other measures of social media sophistication to get a sense of the strengths and weaknesses of what’s currently available. Here are some initial thoughts on who and what a tool to measure social media sophistication should try to assess, as well as things it shouldn’t focus on:
The focus should be on two types of general users. The first group is external audiences like a commercial enterprise’s target market, a government’s constituents, and the key stakeholders for non-profit, healthcare and academic institutions. The second group is people who leverage tools and platforms for professional purposes, both for themselves and on behalf of an organization (e.g., brand and employee advocates, business development professionals).
The assessment should not try to assess social media expertise, particularly in the context of marketing and related usage. Questions like What is the average cost per like for a sponsored post on Facebook?, How many tweets can you send on Twitter per day?, and What mathematical equation describes the amplification you get on social media? reflect advanced topics that only professionals who are engaged with social media full time should be expected to know.
The assessment should focus on public social media platforms, tools, and apps, not private offerings or other digital platforms. With respect to Google, for example, questions could address Google+ and/or YouTube but not the search engine itself or Google Analytics. Similarly, questions about internal social media platforms like Yammer, Chatter, and Slack are probably better suited for a digital literacy assessment, as they are less likely to be universally known or used.
The core assessment should include universal topics. There should be a core set of questions/tests that address topics virtually anyone in the world can answer (e.g., YouTube, Facebook). Similarly, to the extent possible the assessment should tap into concepts, knowledge and usage that apply to multiple platforms and reflect transferable knowledge and skills, even if the questions are platform specific (e.g., the notion of a status update, what a hashtag is/does, the concept of a wiki).
There may also need to be customized questions related to platforms, tools and apps that are primarily used in specific countries (e.g., Sina Weibo and VK).
The assessment should not measure superficial and arcane knowledge. The least valuable kinds of questions to include in a tool to measure social media sophistication focus on:
- Trivia (e.g., the names of Google’s founders, where ChatRoulette originated, who was the only person with a perfect “100″ Klout score until 2013)
- Time sensitive information (e.g., whether Twitter or Instagram has more active users, how many active users there are on SnapChat, what the Fail Whale is)
- Technical terms that aren’t relevant to general users (e.g., what RSS literally means), or device-specific info (e.g., whether flash works on mobile devices)
- History (e.g., when YouTube was started, when Instagram launched video)
These kinds of questions are interesting and including a few of them may have some value, but they should be used in moderation relative to more substantive assessments.
As always, I welcome your thoughts. Do you think there is value in attempting to create a tool to measure social media sophistication? Can it be done? Should it be done? If so, what are the best approaches to creating, validating and implementing the assessment?
Even if we don’t create a measure – which may be unrealistic and impractical – exploring these ideas will enrich our understanding of what social media sophistication is and how we can best assess it.