How To Prevent Another Phoenix Pay System Inferno

An IT system called Phoenix was meant to replace old payroll systems in Canada’s federal government. They estimated it would cost $5.7 million. Then $300 million. A decade later, $4.2 billion has been spent and Phoenix still throws more errors than the systems it was meant to replace.

While politicians continue to throw money into the fire, Canada’s federal employees have been underpaid, overpaid, or not paid at all. Some have lost their life savings, their home, their marriage, their dignity. It’s a catastrophe with no end in sight.

In my book, I chronicle the entire history of Canada’s Phoenix nightmare and offer expert opinion about possible solutions. With the book already in stores, I publish recent info and new perspectives at this website.

Another Phoenix Catastrophe Is Inevitable

According to this public dataset, Canada’s government has nearly 50 IT projects underway that cost more than $100 million. 3 of which cost more than a half-billion. And as the saying goes: the larger they are — the harder they fall.

Whether we investigate IT system bugs, procedural flaws, human error, or IT project funding and governance, it appears Canada’s federal bureaucrats have not learned anything from Phoenix. They are stuck in a state of either deja vu or selective amnesia.

Canada’s government must first disabuse themselves of the terrible assumptions that led to Phoenix:

1. Treasury Board of Canada still thinks the “traditional waterfall” (that’s what they call it) is an optimal way to govern the funding for IT development.
2. The Public Works and Government Services (PWGSC) department still thinks a centralized Pay Centre is an optimal way to manage staff who administer the 80,000+ regulations across multiple collective bargaining agreements.
3. PWGSC is still addicted to the idea that giant IT systems can be wholesale replaced by new, off-the-shelf systems — and Treasury Board’s procurement strategy enables their addiction.

Until those assumptions are dislodged, Canada’s government cannot set a new course based on better principles.

For example, with regard to #3 above, notice Phoenix did not replace the old systems — many of the Regional Pay Systems (RPS) remain to this day! Canada’s government finds itself in the very situation it wanted to avoid: it is simultaneously maintaining multiple, half-baked systems. And they have another on the way: the “Next Gen” product that they started rolling out in 2021.

How To Prevent The Next Phoenix Inferno

Principle 1: Rethink policy and incentives.

The idea that colossal overhauls can succeed is based on faith and wishful thinking, not fact.

It is crucial that a transformative shift in policy take place within the Treasury Board. The government must recognize the pitfalls of large-scale IT system replacement projects and disincentivize such endeavors. Instead, policies must support smaller, more manageable initiatives that address acute problems in the near term and incrementally solve chronic problems toward a cohesive, well-designed future state.

Principle 2: Embrace decentralization.

The notion that all payroll advisors need to be co-located in a centralized office using a monolithic system is obsolete. Two compelling reasons should drive this paradigm shift:

• First, the success of payroll advisors working in collaborative pods is evident. Through 2016 – 2023, there were periods of high velocity through the backlog of payroll errors — in each of those periods, we find that small cross-functional task forces had formed to swiftly address and rectify a high volume of payroll errors. This approach should become the new norm.

• Secondly, there are 80,000+ regulations across the many collective bargaining agreements. It is overwhelming for a centralized office of advisors to master. Pods of payroll advisors assigned to specific sets of collective bargaining agreements can emerge.

Principle 3: Embrace a composable system architecture.

Learn from the past, I say! Canada’s government has run the same experiment repeatedly: buy a commercial-off-the-shelf system and pay a multi-national consulting firm to install it. The catastrophic results of that experiment are well known. (Not convinced? Buy my book.)

A new experiment is warranted. Rather than commit Canada’s taxpayers to giant replacement projects every 5 years (collosal failures, all of them), let us acknowledge that for as long as the federal government exists it must issue paycheques every two weeks. The system (i.e., the individuals and their interactions and processes and tools) must optimize for maintainability over time.

Let’s prioritize interoperability, composability, and adaptability. Technologies and engineering patterns that emphasize these principles are now at the forefront, rendering the goal of a single, centralized, commercial-off-the-shelf system obsolete. A modular and agile approach to system architecture would better align with the ever-changing needs of the workforce.

Principle 4: Foster agile engineering practices and an evidence-based management culture.

Let’s shed the counterproductive belief that external, multi-national companies hold a monopoly on intelligence and capability. It is time for Canada’s government to cultivate an agile engineering culture within its own ranks.

Internal teams can be organized around the iterative and incremental evolution of the IT systems and business processes. And, to be clear, I’m talking about teams composed of internal staff combined with external consultants. Internal staff are generally motivated toward smooth operation and long-term goals but can think too narrowly and become insular; consultants ought to be strategically employed to invoke fresh ideas and spur innovation.

Principle 5: Do the hardest part extremely well.

I respect that digital architecture and software engineering is complex. But let’s be honest: we’re talking about straight-forward math, age-old accounting practices, and formulaic calculations. In terms that a computer science graduate can understand: we’re talking about basic computation, storage, and networking. This is not the hardest part of the Phoenix problem!

The most difficult challenge relates to the User Experience.

We know by parliamentary committee evidence that it takes up to two years to train and onboard new payroll advisors. Their challenge is twofold:

1. 80,000+ regulations across the many collective bargaining units.
2. Learning how to navigate and use clunky and cryptic software interfaces.

#1 can be resolved, in part, using the pods I mentioned earlier. But #2 is an area where great service design, business process design, and user interface design (UI/UX) can be transformative for Canada’s government.

Anyone who has tried to submit invoices through SAP’s Fieldglass, or enter a timesheet in Oracle’s PeopleSoft can relate. The interfaces are complicated, visually overwhelming, difficult to comprehend and common tasks often take many clicks across many screens — it’s no wonder the users of Phoenix Pay System are so error prone.

The proximity between software designers who serve the pods of payroll advisors must be closed. Design is not a one and done activity, it is ongoing. A critical feedback loop must be established between designers/developers and the users they serve. Imagine if payroll advisors can swiftly comprehend the screens they use, navigate rapidly through pay processes, and feel confident using the digital tool.

The Phoenix catastrophe is unique only in that it couldn’t be swept under the rug — it’s highly visible because it affects so many employees. But the sun will eventually set on the aftermath of the Phoenix Pay System. (I forecast another decade.) But Canada’s government will continue to conduct giant IT projects — and many will be collosal failures unless management methods are fundamentally changed. I believe the principles I’ve outlined are pragmatic and can inform a strategy to reduce risk and improve outcomes.