Turbocharging Commerce for 50 Million Viewers

An inside look at the custom graph database and business rules engine scaling personalized offers for millions

Sky is one of the world's leading entertainment brands, serving over 50 million customers globally. This case study offers a behind-the-scenes view into a custom in-memory graph database and business rules engine developed in Python. Powering the core pricing and offers across Sky's streaming brands, this innovative solution enabled granular personalization and optimization while keeping complexity manageable.

We detail the technical decisions and tradeoffs made to balance flexibility, performance and time-to-market. Additionally, we explore the sizable business impact of putting data-driven offers in the hands of customers.

The Need for Speed and Agility

Sky found itself at a transformational point in its business model. The rapid shift from satellite broadcasts to on-demand streaming meant that legacy systems were no longer able to react quickly enough to marketplace demands. Facing competition from global players, Sky needed to accelerate innovation in how they matched content to viewers.

At the heart of this mission-critical effort was building intelligent, data-driven engines for pricing bundles and offers. However, product managers demanded flexibility to run constant experiments and change business rules on the fly based on customer data. With a global audience of over 50 million diverse users, Sky's platforms also required snappy performance and bulletproof reliability even during peak demand.

Tying this all together was an intricate web of eligibility rules, discounts, regulatory controls and dependencies between various products and territories. Hard-coded packages were no longer an option. The technical challenge was building a sophisticated real-time business rules engine tailored to Sky's needs that could balance all these constraints.

The existing on-premise infrastructure built up over years via acquisitions was fragmented and ill-equipped to adapt at the pace required. Teams were slowed down by legacy code and dated architecture. Quick iteration was hampered by bottlenecks across siloed systems, inconsistent data flows and manual processes. It became clear that innovating would require starting from the ground up with greenfield development optimized for the unique demands.

With personalized pricing experiences flagged as a top driver of conversions and loyalty, our small but nimble team took on this undertaking. We had to build for flexibility first while innovating on performance and scalability. The answer would come in the form of a bespoke in-memory graph database and business rules engine tailored precisely to Sky's product domain.

Building a Custom High-Performance Graph

It was clear from the onset that off-the-shelf solutions would struggle to scale to Sky's demands while retaining the flexibility critical for business users. So we made the unconventional decision to build a highly-optimized custom engine specifically for pricing Sky's products across all routes to market.

At the core of our solution was modeling Sky's product catalog, discounts, bundles and associated rules as an in-memory directed acyclic graph (DAG) in Python. Each product, offer or rule in the system became a node populated with relevant attributes and metadata. Edges linking nodes encoded the multifaceted relationships and restrictions such as hierarchies, exclusions, prerequisites and more.

This schema-less graph structure gave us flexibility to rapidly incorporate new data-driven elements without costly migrations. We could tweak node contents and connections on the fly to address changing needs. Python allowed quick iterations informed by user feedback. (We started with Python 2.7 and had to migrate to Python 3 half-way through the project, which was a mini challenge on its own - but brought about welcome refactorings and code quality improvements.)

Of course, flexibility alone does not translate to performance at scale. We put heavy emphasis on optimizing data flows and access patterns tailored to pricing operations. Carefully layered indices (mostly in the form of look-up dicts) gave snappy traversal through thousands of graph paths necessary for evaluating eligibility. A Python caching layer led to the majority of read traffic served out of memory.

By combining multiple bespoke graph algorithms and heuristics, we helped the business optimise offers for customer lifetime value rather than single conversions.

The end result after months of focused iteration was a pricing engine capable of sub-second response times even with the trickiest customer baskets and offers. But more importantly, we built an agile foundation tailored to Sky's products that balanced performance with the ability to quickly test and update business levers.

Empowering Business Users through Data-driven Abstractions

A pillar of our solution was enabling business teams to tweak offers and rules without engineering bottlenecks. We knew that pairing data science with human domain expertise was key to long term success.

To achieve this, the business invested in an intuitive visual interface that formed an abstraction layer on top of the complex pricing graph. Through this tool, product managers could model decision logic by manipulating nodes and edges representing the various products, discounts, requirements and relationships.

Under the hood, the tool automatically handled translating user-friendly objects into properly formatted data for the pricing engine to consume. Exported to XML and fed directly into the Catalgoue Services component, the modeled nodes, edges and their attributes map directly to our bespoke Python data structures, like a fast-acting in-memory injection.

Version control mechanisms ensured changes could be rolled back easily without risk. Validation checks helped business users build reliable logic, running simulations based on real customer data. We focused on surfacing the most valuable performance metrics and KPIs to enable data-informed experiments.

What this meant was product experts now possessed the capabilities to freely adjust catalog contents, personalized discounts and offer rules at a rapid pace. Empowered teams moved faster and smarter, leveraging data to optimize campaigns and retention initiatives. They could roll out experimental offers and quickly walk back failed tests without being gated by software release cycles.

Even more convenient was the fact multiple brands across different markets could share the same flexible catalog representation. This allowed Sky to pursue global pricing strategy harmonization efforts and maximize the value of its worldwide entertainment portfolio with reduced duplication.

By abstracting away infrastructure complexity behind purpose-built tools, we successfully put power directly in the hands of business decision makers. This was the final piece enabling Sky to move at the speed demanded by modern viewers.

Driving Growth through Personalized Offers

The bespoke pricing engine and continued optimisation from empowered product teams provided tangible business value on multiple fronts.

Most importantly, Sky could now tap into the power of data-driven personalized offers tailored to the taste, preferences and willingness-to-pay of customers across global regions and markets. Matching viewers to entertainment packages that delighted them in an intelligent way led to material lift across crucial metrics:

• Conversion rates on promotional churn offers increased by over 20%
• Clickthrough rates on upsell campaigns improved by nearly 40%
• Customer lifetime value rose steadily over 3 years for subscribers exposed to personalised pricing

Interestingly, the flexibility to run ongoing experiments combined with rapid iteration based on observations meant the engine kept getting smarter. The graph allowed easy incorporation of new data signals - streaming habits, broader customer lifetime journeys and macroeconomic indicators - to continue refining accuracy of recommendations.

And with trustworthy abstraction layers, Sky's product experts could leverage these capabilities with minimal friction. Our custom solution enabled them to focus energy on high-value problems like maximizing customer joy and iterating on creative initiatives rather than wrestling with infrastructure.

With the new pricing engine humming as a competitive advantage, the business turned focus to global subscriber growth and expansion. Bolstered by the success powering 50M+ Sky customers with personalized recommendations, our platform was soon re-engineered to also serve Sky's streaming offshoot NOW TV across Europe and Asia as well.

By maintaining sharp focus on domain-driven development tailored to Sky's specific needs, we built an advanced commerce solution ready for the realities of modern entertainment. Our custom graph engine helped unlock enough flexibility, performance and automation to let Sky innovate faster than ever.