MODEL RISK MANAGEMENT:
How to Avoid an Earnings Surprise
The industry is
going down the
mark-to-market route,
creating significant
opportunities for
earnings swings
and distortions.
December 2004
By John Bampfylde and David Shimko
Model risk is no longer the purview of quants and analysis groups hidden deep within the organization. With mark-to-market (MTM) accounting, the quants' MTM pricing models now drive earnings, and, as we all know, earnings drive CEOs. Model risk was dragged into the spotlight with the 2002 and 2003 restatement of earnings by many energy-trading concerns, due in large part to over-optimistic model valuations, which removed billions of dollars from the industry's balance sheet. These losses were far in excess of any value-at-risk (VAR) limits, causing stock prices and investor confidence to plummet. Given that many utilities as a matter of necessity must trade in energy markets and new financial traders also are playing this market, the issue of model risk is again a cause for concern.
Furthermore, the triple-whammy of MTM accounting, market illiquidity, and substantial legal liability for auditors massively increases model risk in the electricity industry. Many participants focus on market and credit risk, which is a perfectly reasonable approach, but in some cases it is to the exclusion of model risk management. This exclusion may cause undue risk to future earnings.
Model Risk, MTM, and Mark-to-Model
What is model risk? A model can be defined as a mathematical representation of price relationships in the marketplace. Model risk occurs when either the model does not represent the market accurately or when it may not appear to do so; both present big risks. If the model does not represent the market accurately, the contract will be misvalued and earnings will be misstated. If the model appears to be inaccurate, auditors and investors may apply a higher uncertainty premium to those claimed earnings.
Model risk exists when a model stands between a market price and the valuation of a financial product. Common market indexes include exchanges (e.g., NYMEX and ICE), various broker quotes, and index-reporting organizations.
Most traded products do not fit exactly into the index categories, however. Experience shows that "plain vanilla" hedge contracts, as priced in the market indexes, often are relatively ineffective hedges for a generator or retailer (in particular). Thus, utilities seek out non-standard, or exotic, products that may be more effective hedges and perhaps better value. Examples include monthly contracts, structured transactions, options, non-firm contracts, and combinations thereof.
Energy traders model these exotic contracts as part of their core business and buy and sell based on the model's values; this is the business of trading. The problem arises when MTM accounting is applied. Most companies keep the exotic contract either until delivery or for a long period of time, since they are non-standard and therefore difficult to on-sell. Under MTM though, these contracts have to be frequently valued to get the current liquidation value. The previously applied valuation model is the most obvious way to calculate this MTM.
This is "mark-to-market-through-a-model," or mark-to-model, where the model's output is taken almost directly to the profit and loss (P&L). This is a risk because even if the model is a perfect representation of the market, others may not see it that way. Auditors will require proof that the model is perfect before they approve the accounts. Aggrieved shareholders could sue the company claiming the imperfect model distorted earnings and hence share price. Not only does the model have to be perfect, it has to be seen as perfect.
But anyone who has traded in illiquid markets knows there is no such thing as a perfect model. An element of "black art" or common sense is applied to all valuations. Model risk management is where these assumptions and the model's logic are documented and tested so that the auditor and the aggrieved shareholder's legal team can clearly see that the company has been reasonable and conservative with regard to calculating MTM earnings through internal models.
Learning From History
The 2001/2002 crash of many energy traders' stock prices, the many restatements of earnings, and the subsequent lawsuits provide good examples for how not to do MTM, as well as some shining examples of probity.
Examples abound from the days of energy marketing in the late 1990s and early 2000s, when valuations were less rigorously audited. A marketer would do a deal and mark it to market immediately to create instant "earnings" of $10 million, $50 million, or even $100 million (yes, really); in some instances the counterparty on the other side of the deal reported a large profit too, if that's what their model said. This tendency was exacerbated by the traders' performance-related pay, with the "gamers" creating high unrealized "earnings" and collecting an employee performance bonus of cold hard cash before the music stopped, so to speak.
Companies have, and had, two approaches to this problem. First, they apply operational risk management and effort to ensure the model is a reasonable representation of real life (i.e., the output values are as close as possible to actual market values). This is an ongoing process. As more price data is observed, models can be modified to incorporate these observations.
For example, after a few years of observing out-of-the-money option values, a volatility "smile" can be incorporated into the valuation process to capture the observed market value above the theoretical value. The first guiding principle is a simple check: At the time of the deal the most likely MTM value is slightly negative, a reflection of the bid-offer spread.
Second, traders have well-developed procedures to track and document how the model works, what changes have been done to it, and how these changes have altered the output values. This documentation improves transparency of operation but, more importantly, is a fundamental part of the auditing process and any legal defense. A lack of documentation and procedure plays right into the hands of plaintiffs and creates a big risk for an auditor.
Model Risk Management
Model risk management, therefore, is the process of applying operational risk management to modelling and valuation models to demonstrate that the models are robust and the assumptions reasonable and explicit. Good model risk management requires the same discipline and attention that companies give to market risk management.
First, companies need a policy to dictate the minimum acceptable standards for all models. Models that allocate capital and directly impact earnings require the highest standards, but almost all models in an organization require a minimum standard since, by definition, the models help with business decisions. The policy should describe the conventions of model development, ownership and management of models, model structure conventions, acceptance testing, the sign-off process, and change management. The same policy standards and gravitas that are applied to a company's VAR calculation should be applied to all valuation process models.
Second, each model should be thoroughly documented, ideally during the building process. Documentation should describe the model's objective, the algorithms, and the inputs and their respective units. Documentation also should provide a user manual. The intention is that anyone provided with this documentation can use the model, a key part of personnel risk management.
Third, the model should be tested to ensure it functions as required and as defined in the documentation. By this stage the model should have been tidied up to remove unnecessary code and to rationalize its operation and features to make it more company-standardized. The model should be accepted as fit for use only after successful testing. Then the model should be frozen, preventing any changes from inadvertently altering the model's output. User passwords and password protection of code are two common tools employed.
Fourth, each model should include as an integral part of the code a written commentary describing its version number and date, the developer/manager, and any changes or bugs found since acceptance.
Fifth, given the serious financial consequences of a flawed model, key models should be reviewed or audited by external parties. There are two broad types of review (and acceptance testing): white box and black box. White box testing is line-by-line checking to ensure that the model uses the defined algorithms as intended, and that the outputs are correct. Black box testing is a benchmarking/calibration process, not looking so much at the formulae but how the model output compares to what is expected. Given the complexity of many portfolio and risk models, black box testing often is the easier process. It also limits the potential for losing intellectual property in the process.
External review is a plug for specialist consulting services. Many companies don't recognize that external review is also a fixed cost. Since auditors have been trampled by not spotting earnings manipulation, and they now spend a lot of time making sure they understand exactly what they are auditing. The client pays for this time, either directly through the auditor or by more proactively managing the audit through specialist model review and sign off.
Corporate Action
The Financial Accounting Standards Board's statement 133 and its international counterpart, International Accounting Standards statement 39, have pushed the energy industry down the MTM route, which creates significant opportunities for earnings swings and distortions. To avoid this distortion or the appearance of distortion, companies need to apply model risk management where a model creates a value that goes directly to the P&L.
Energy marketers' experience shows that common sense can go a long way to limit model risk (i.e., "fast" MTM earnings are a rare occurrence and are more likely a reflection of mis-marking a deal). Common sense needs to be linked to a structured model risk management program to ensure models maintain integrity and to leave an audit trail of assumptions and changes.
Model risk grows exponentially where MTM accounting, illiquid markets, and complex products combine. These three features combine in power markets to create the perfect storm for model risk. When auditor liability is thrown into this storm, it generates a large, unavoidable cost that can be minimized only through dedicated corporate model risk-management action.
John Bampfylde and David Shimko are partners at Risk Capital, an independent risk management consultancy based in New York. Contact Shimko at dshimko@riskcapital.com.
Articles found on this page are available to subscribers only. For more information about obtaining a username and password, please call our Customer Service Department at 1-800-368-5001.
Public
Utilities Fortnightly & Spark
