Ins
and outs of software testing for Communication Service
Providers
By Isabelle Paradis, HOT TELECOM
July 2012
Getting
testing right can make a big difference to both
revenue and cost and many of the CSPs we spoke to
during our interview process constantly referred
to the need to reduce costs and the time to market
for new communications products.
Testing
service providers outlined how this can be achieved,
for instance, through centralization and off-shoring
of some testing functions, placing a greater emphasis
on requirements planning before test planning and
execution begins, and critically, by reducing the
rates of software defects in the production environment
(that is the real operating environment rather than
the controlled “test environment”).
Based on our discussions with testing service providers,
it is especially important to reduce errors in those
systems that have the most direct impact on a CSP’s
financial and operational performance.
Darren
Coupland of Capgemini-Sogeti’s Global Testing
Team, and currently leading a major testing and
ALD transformation program for a tier one telco
player comments: “It’s no longer
good enough to just detect defects during the test
phase and prevent production incidents; that is
a given. Increasingly CSPs now demand that test
providers really focus on stopping business-critical
defects being introduced in the first place by taking
a Shift Left approach. This isn’t anything
new, but it’s always been difficult for test
teams to generate the necessary traction and influence
‘upstream’ changes. But with a current
client, we have developed test transformation programs,
with quality improvement at their core, that combine
advanced governance, metrics and accelerators, and
our own thought leadership & IP, with aim of
not just transforming the testing service but also
the quality of wider project deliverables that have
a direct impact on the client’s business goals.”
When
CSPs were asked to prioritise the importance of
systems to be tested, billing emerged as the most
important system to test thoroughly (see table 1).
Around half of CSP respondents singled out the billing
system as most critical from a test perspective.
Critical
software applications, such as the billing system,
can be a source of major headaches for CSPs: functional
limitations, or the reconfiguration effort required
to support changes to products or the introduction
of new products, mean that time-to-market can be
long: testing is an intrinsic part of reconfiguration
or functionality improvement projects.
Table
1: Systems described as “most important to
test” by CSP interviewees
Source: HOT TELECOM’s Software Testing for
CSPs market analysis
Nevertheless,
some CSPs told us they believed that no one system
was more significant in testing terms than any other.
Rather, it was more important to conduct system
integration and end-to-end process testing, as the
greatest number of errors tends to occur in the
gaps between individual systems.
From
the discussions we had with CSPs in relations to
how they measure the success of a testing program,
we can conclude that CSPs take different approaches
to measuring success. Some have a set of very detailed
metrics that cover all aspects of a test process,
while others simply say that reducing the number
of software errors going into the live production
environment is the only significant measure of success.
Table 2 indicates the ranking of those measures
mentioned to us by interviewees.
Table
2: Metrics used to judge the success of testing
programmes
.
Source: HOT TELECOM’s Software Testing for
CSPs market analysis
Many
of the test managers and software quality assurance
specialists that were interviewed said that software
testing was increasing in importance and becoming
more challenging, and that the ways in which it
was carried out were changing, in response to a
number of specific drivers:
The
four key drivers which were identified through our
discussions with CSPs, increasingly impacting their
telecom software testing needs are:
