It’s not uncommon to come across ‘become data driven’ as an annual objective, whether at a fledgling start-ups or a public sector institution. In these cases, it’s often positioned as an end, rather than a means.
But there’s a risk to seeing data as an end goal; having data for the sake of it does not always lead to better decision making. There are some instances where it is better to be without data than with it – where the data you’re using is inaccurate, or being incorrectly interpreted, or if it’s not used at all in decision making.
Given that expense, collecting data should be approached like any product – in iterations and working from a solid, user-first specification. With my product manager hat on, here’s how I’d avoid ending up with an unhelpful, unwieldy set of data.
Make sure it’s needed
A CTO I worked with used to meet any request for more data with: “Will it make you more accurate than your best guess? Does it need to be more accurate than that?” His point was to start out with whether there was actually an issue with taking our best guess – in many cases, this was a valid choice. More data is only of use if it’s likely you will make a different decision, or take a different action, to the one you’re making now.
It’s not just a question of whether more data is needed, but also how much. All I’d offer here is that, while it needs to be decently representative to be of use, you rarely need everything. It’s worth voicing and agreeing with the team what you consider to be representative, and how much margin of error you’re comfortable with. If you do decide to gather it, make sure you’ve made a conscious decision about how long you need it for, too.
The exception to this is if you are working on predictive models, or any machine learning. In this case, go broad and go big.
Make sure it’s accurate
There’s often a sense that data is inherently more accurate than judgement calls. But, as we know, wherever humans are involved – in this case, in the input and analysis – there’s room for error.
If you’re collecting data anywhere under your control, whether it’s your website or your CRM, don’t make it difficult for users to input accurate information. Use human-friendly labels to reduce the risk of a user choosing the wrong category or typing in a number in the wrong unit. And banish 47-option dropdowns – sales representatives or call centre agents will rapidly cease to pick anything other than the first five they remember.
If you’re going to make a field required, you probably need an ‘Other’ option, preferably with a free text addition, to avoid people just picking what’s closest. If you get too many, do a review of the free text and create a new option if it makes a difference to your decision making. Put in a calendar reminder to cleanse your dropdowns every quarter or so – last year, YouTube ads might have been a common response to ‘How did you hear about us?’, but if you’ve stopped advertising there, the option isn’t needed anymore.
Where you have internal staff inputting the information, involve them in what it’s used for. Too often the data is entered, never to be heard of again, and inevitably that decreases staff’s motivation to do so correctly. I’m a big fan of back-of-the-toilet-door posters and ad hoc emoji-filled Slack announcements for this – go to where the people are.
Make sure it’s used
As with any product, data is only as useful as how much it’s used. So often, companies undertake huge data collection or visualisation projects only to find that Tableau is only opened once a month for the sales report, and no one can remember the password.
If you’re lucky enough to be starting from scratch, involve non-techies in your data modelling. Words that are logical to a tech team are rarely those which come most intuitively to the rest of the business. I’ve seen hours of work on reports wasted when the results showed that ‘order’, which I’d taken to mean X, actually meant Y. Keep a glossary of the data terms and relations in a format that does not scare non-techies – Github may be great for openness but it’s off-putting for many.
When it comes to third party data visualisation tools, there’s a suite to choose from and often these will be inherited from whatever ecosystem the organisation has invested in. If you do have a choice, I’ve found it’s most successful to use whichever tool the majority are familiar with.
If data isn’t being consumed as readily as you were hoping, or there’s been pushback, avoid swapping visualisation tools as if a new one will be the magic pill. First look at which questions people say the data can’t answer, or what’s taking more time than it should. It may be that a data modelling decision is preventing clear connections – correcting that may be quicker and cheaper than purchasing new software. Before committing to a tool, use the free trial to recreate existing essential reports, answer an outstanding question and combine multiple data sources.
If you introduce new data visualisation software, it’s not enough to ask business users to ‘have a play around’. No one prioritises this kind of vague command. I see it as the responsibility of those introducing the tool to make sure it’s usable and used. I tend to set each data user a specific, real business question to answer (with a deadline), and a task related to someone else’s report. Being able to collaborate on data visualisation and analysis is half the battle.
Making sure it’s helping
Once it’s there, data is often left to age without any checks on its ongoing usefulness or relevance. Use analytics to see if the data visualisation and/or reports are being opened and used. If you’ve introduced a new tool or data set, schedule a check-in 6 months on to assess usage and examine what more is needed and, as importantly, what can be taken away. Ask yourselves: how has it helped? And what has it changed?
Collecting and manipulating data can be an incredible time sink. I’ve watched months go by while reports are set up, only to see them left unused, or cause more arguments than they resolve due to worries around accuracy. I think it’s because businesses think about data as a one-time-thing that can be completed and then you ‘have’ it. But if it can be re-framed to a shifting thing that needs planning, iterating and communicating – to a product that requires product management – I think the project can be altogether more successful.
Impact beyond Alpha
4-8 weeks of a Discovery or Alpha project is a fleeting time for us to accomplish our mission: to help institutions embrace the opportunities of the internet era. What can we try to extend our impact beyond the confines of the project.
Personal principles for product in a crisis
Building at pace for BEIS
Our Notbinary squad has been working with BEIS to build public services to meet the requirements of the Coronavirus (COVID-19) outbreak.