Internal and External Data
At last, we've arrived at where we actually find the data. The short answer is, obviously, everywhere.
It may be more appropriate to talk about what kind of access and ownership you have to various datasets, and how you can go about making them available and utilising them. To start with the latter, we must first get acquainted with the term “data silos”.
Data silos
The good news is that all businesses have loads of data. The bad news is that they are most likely scattered all over the place.
Perhaps you’ll find something you need in an email attachment, while some other data is located in various cloud solutions or a warehouse full of paper receipts that no one has had time to digitise yet. Or it is distributed across various applications, on local servers, in the cloud, in email attachments, on your desktop and with suppliers.
When data is separated in this way, we say it is found in different “silos”. As long as the data remains in silos, it's very difficult to utilise, or to compare information from one data silo with that found elsewhere. We therefore need to lift our data out of the silos to derive meaning and value.
Example
Bike rentals
A business that rents out electric bikes has data about who rents bikes, at what times, for how long, and at what prices.
In addition, they may be interested in third-party, external data about weather, traffic, public transport, working hours, sports and cultural events, where and how people move, and so on.
By collecting several of these data, they can discover correlations and insights that help them work more efficiently and earn more money. For example, joining these types of datasets together might help them to work out where they should place their bikes in order to attract the greatest number of customers.
To do this, the business would first need to gather and make all the relevant data available in one place—in this case, perhaps a cloud-based database platform—in a format and a structure that allows the data to be used together. Where and how the data can be structured and stored, you will learn more about later.
Once the data has been lifted out of these various silos and gathered in one location, the company now has much better opportunities for analysing the data. They can now follow likely usage patterns and demand for bikes in their data, which can help them to decide where and when they distribute bikes. They might even discover opportunities for new business opportunities and strategic collaborations … perhaps they can find a gelato shop that a subset of their customers ride their bikes to in hot weather and offer an interesting advertising opportunity to them?
Insight
What does it mean to make data available?
Making data available can in some cases be as simple as mapping where it exists, and ensuring that those who need it have the correct login details, and access permissions etc.. But the process of lifting the data out of the silos is also important. It could simply mean downloading data from various sources and manually collecting them in one place, such as in an Excel spreadsheet. This is completely fine if, for example, you are looking for a specific insight and are making a report.
However, when we talk about data-driven businesses, such manual work will not be adequate. Instead, you will need to set up some form of automation in order to collect larger quantities of high quality data. This might include for example, using APIs to collect all relevant data in one place. What kind of “place” this can be—which could include databases and other forms of data platforms—you will learn more about in the next chapter. There you will also see that the data often needs to be processed and cleaned up before it is usable by data analysts.
What access do we have to the data?
For a business or organisation, relevant data will exist both inside and outside the business, that is, the data can be internal and external. This distinction is important because it concerns the ownership of the data, which affects both access to them and how easy they are to use.
A useful way to sort our data sources is by our level of access. If you are conscious of this, it is also easier to know what you can do with the data and how it can be processed in later stages.
Internal data may, for example, be collected by or shared with customers, partners or suppliers and not be yours alone, or its use may be subject to laws and regulations, for example relating to privacy.
With this in mind, let's now look at where you can get data and what access you have to it.
Internal data
We can distinguish between two types of internal data:
- Internal, unshared: Data that the business itself has collected and/or controls, for example, spreadsheets about accounting, customers, logistics, operations, equipment, etc.
- Shared data: Data the business completely or partially controls and have access to, but which others also have ownership of or access to. In this case the business cannot necessarily change or use the data freely. For example, a gym member profile might contain personal information about a person's gym activities at the club itself, as well as payments and contact with the customer service office that are stored in a financial system that is owned and operated by another company that they have procured this service through.
All businesses have internal data, but the volume, quality, and availability may vary enormously. Here are some of the categories (but far from all):
- Customers: Names, addresses, emails, demographics, locations, business registration numbers, customer service interactions, history, potential customers
- Website: User behaviour, referrals, dropouts before purchase, search engine optimisation
- Social media: Target audiences, adverts, results, engagement, spending
- Supply chains: Supplier data, prices, inventory, logistics, transport, orders, potential suppliers
- Market / industry: Competitors, prices, trends, search trends, surveys, related products and services, business models
- Operations: Sensors, control systems, equipment data, maintenance data
- Accounting and transactions: Purchases, sales, fixed expenses, fixed income, other expenses and income
This is by no means an exhaustive list, but there may be examples here that you recognise?
The tinsmith’s workshop
Let's look at an example of what kind of data a business might have internally—in this case, the fictitious workshop “The Tin Can”.
But even if we become really good at using internal data for innovation and improvement, there may be as much or more to gain by looking outward—with external data.
External data
Again, we can make a distinction between two types of external data:
- Open/public data: Data from external sources that is openly available. By “public”, we do not necessarily mean that the data comes from the public sector, but rather that it is available to the public.
- Proprietary/private data: Data from external sources that not everyone has access to or permission to use. Such data may come from suppliers and others in the value chain of a business, or it may come from third-parties who collect and sell data.
External data is data we do not own and control ourselves, but which we either find openly available or may gain access to via partners and suppliers.
This can be data about things like traffic and weather—or data about production, logistics, maintenance and process at other businesses with which we exchange data.
Just imagine how many specialised businesses are involved when, for example, a ship is being built or how many entities are connected to the export of goods and resources.
In these scenarios, many businesses are working towards the same goals and can benefit from cooperation and data exchange—whether it's to reduce emissions, deliver better services or to build smarter, more sustainable cities and communities.
External data can often be integrated directly into a business' own systems and services through APIs, gaining even greater value.
Typical examples of public data are data from the governmental agencies, such as the Brønnøysund Register Centre (Norwegian public registres) or the Norwegian Mapping Authority. Here there’s tons of data that anyone can access. Organisations like the World Health Organisation (WHO) and UNICEF, also share large amounts of data that can be freely used.
There are also some public datasets that are not immediately openly available, however, such as certain information from the Population Register or the Tax Administration—though one can apply for access.
Example
Fiskher
The possibilities with open, external data are best illustrated through a real example. Namely, the popular app Fiskher (stylised form of “fish here” in Norwegian), which shows recreational fishermen all over Scandinavia the way to over 70,000 fishing areas.
The app would have been difficult to create without open data from the Norwegian Mapping Authority and Geonorge, which is freely available for everyone to use.
By combining satellite images, mapping data, weather data, and a number of other datasets with machine learning, it was possible to pinpoint thousands of habitats for fish.
Fishermen can also register their catch, which means that Fiskher produces its own data that again can become useful for others. For example, it gives institutions like the Norwegian Institute of Marine Research and the University of Tromsø a better overview of recreational fishing in the country.
Today data is a significant part of the world economy, and several suppliers of proprietary data are multinational companies with thousands of employees.
Here are some examples of proprietary data:
- Copyright-protected data: Intellectual works such as music, film, photo, novels, textbooks—as well as patented brands and technologies.
- Market research: Professional market research agencies can provide data and insights about specific topics or issues on demand.
- Data enabling digital advertising platforms: Social media platforms like Facebook, Snapchat, and TikTok—but also online newspapers like VG and Dagbladet—own data generated on their platforms. Strict privacy rules prevent the outlets from sharing or selling detailed personal information. However, compiled user and behaviour data can still be used to create tailored ads for specific target groups.
If you are a leader or employee in a business—or even if you just have a small pet project—it might be wise to stop and think about what external data you could benefit from, and what data you have that others could benefit from.
But remember: Even if you own data yourself or have purchased access, it doesn’t necessarily mean that the data can be used freely. You still have to comply with laws and regulations like the GDPR. More on that shortly.
But first, we have to touch on something we've skipped over so far: Namely, that collecting data means we need to collect it somewhere. The data must be stored. We'll quickly look at this in the next topic.