Development of a Data consolidation and analytics platform for a US analytical agency

Development of a Data consolidation and analytics platform for a US analytical agency

Client

Our client is a US analytical agency that collects their customer-companies’ sales & marketing data (e.g. subscriptions, revenue, customers, pricing metrics), user behavior data (pageviews, articles, conversions, and other actions on the site), then prepares analytical reports as well as forecasts to increase target KPIs.

They operate across various business domains: Telecom, Retail & eCommerce, Media and Entertainment, Education, Fintech, Healthcare, HVAC, Sports, and many more.

Challenge

The demand for the client’s services was increasing number of projects. However, the processes of project onboarding and management — including accessing, collecting, and analyzing customer data, preparing reports, and choosing scripts/algorithms for data processing — were completely manual.
To increase team bandwidth, bypassing recruiting new employees from the local market, the client contacted Lightpoint to develop a custom Cloud-based system to automate a wide range of processes.

Project Description

Part 1: Development of a Self-service portal (SSP)

We developed a web portal that allowed client employees to manage all their customers via a centralized interface, in particular:

  • perform project onboarding
  • setup configuration of data analytics: what data to collect, how to map it, transform and process
  • communicate with customers

Besides the employees, the customers also got access to the interface to track the project progress and make changes to the configuration if needed. That was done to increase customers’ autonomy, promptness when applying changes, and save employees’ time.

The SSP portal was also responsible for producing analytical reports from the imported customer data. The reporting system comprised a catalog of in-built report types — various graphs, charts, schemas, maps, calculations, tables, documents, etc. Calculation and visualization of the outcomes was completely automated. To compare, previously the client employees used Tableau for that, which required a certain degree of manual manipulations.

Part 2: Development of a back-end platform for data collection and processing

Our client works with different businesses from different industries, so data sources (databases, websites, text docs, cloud repositories, etc) and data structures (format, set of fields, datasets, etc) are also varied. Our task was to implement the mechanism that won’t require code rewriting to enable processing the data of each new client, but only configure data mappings to unify all the data coming.

How the final solution we built works:


Phase 1 – Data collection


The system collects data (static files and website traffic) from source systems regularly in accordance with a schedule. Then the data is converted to a common format and structure and undergoes basic enrichment – grouping related data pieces to make them more detailed and descriptive.


Phase 2 – Data processing


Here a deeper degree of data enrichment takes place, now with metrics that are generated via Data Science algorithms. After several cycles of enrichment, the data becomes ready for analysis, reporting, and visualization.

Additionally, we implemented:

  • generation of notifications that inform the user when metric exceeds threshold value
  • generation of recommendations based on predictive analysis
  • calculation of lost revenue


Phase 3 – Calculations for reporting


The system finally combines all the obtained data and prepares regular insights based on it. The SSP portal ingests these insights and visualizes them for end users.

Key features

Centralized user interface for both employees and customers
Automated project onboarding and customer interactions
Completely automated data visualization
Engine for unification of different data structures
Support of data import from various sources

Screenshots

Team Composition & Project Duration

It’s an on-going project and up to now it’s duration is 18 months, the Lightpoint team consists of: 5 Back-end developers, 1 QA Engineer, 4 Data Engineers, 1 Business Analyst, 1 DevOps

Major Tech Stack

.NET, NodeJS, Angular, Golang, Python, Kafka, Postgres, Clickhouse, AWS

Result

The project is long-term with a wide roadmap of further functionally, but the major expected result is automation of data analytics processes so that the business can easily scale with no need to hire internal resources and significant cost savings due to hiring a team from Lightpoint.

Ready to talk about a similar solution for your business?

Get in touch with us at:

    Select a Service:
    Full name
    Email
    Description