Demonstrating data completeness and integrity (DC&I) for mission-critical applications is a necessary step for financial services institutions working to address the anti-money laundering (AML) model validation guidelines issued by the Office of the Comptroller of the Currency (OCC)—"Supervisory Guidance On Model Risk Management" and the Federal Reserve Board as OCC 2011-12 and SR 11-7, respectively (the Guidance).^1,2

DC&I refers to the quality and content of data that is delivered to a model for that model to fulfill its intended purpose. If data is corrupted or compromised in the delivery process, a model cannot operate as intended. DC&I is designed to ensure that complete and accurate data are delivered to models.

The core elements of DC&I are captured in the “Ongoing Monitoring” section of the Guidance. The Guidance states: “Process verification…includes verifying that internal and external data inputs continue to be accurate, complete, consistent with model purpose and design, and of the highest quality available.” To assess DC&I, assessments typically focus on points in the data-flow process where data is extracted, enriched, mapped, transformed and/or loaded. Based on our experiences, financial services institutions face many challenges in comprehensively testing, documenting, and monitoring DC&I for AML and sanctions screening applications. While there are many considerations for proper assessment and maintenance of DC&I, this paper focuses on the following three leading practices for DC&I:

1. Documentation of data and system architecture;
2. Evidence of testing that is clear and detailed; and
3. Comprehensive extract, transform and load (ETL) controls.

The following three leading practices may assist financial institutions not only in maintaining compliance with the Guidance, but also improving an organization’s overall awareness of the AML applications’ data-related components.

Leading Practice #1: Documentation of Data and System Architecture

Many financial institutions have complex infrastructures due to constantly evolving products, customers, data systems and sanctions/AML applications. As a result, it is critical for financial institutions to have a deep understanding of the data and system infrastructure at both a high and low level. Typically, a select few within an organization have a comprehensive understanding of the data and system architecture, and such knowledge is not documented in detail or with completeness. While their systems work and often work well, in the absence of those key individuals, changes cannot be made and specific questions cannot be answered. This type of information is expected to be institutionalized.

Financial institutions should maintain and keep current several pieces of system architecture information. This documentation should be straightforward and clear, in order to inform parties with varying levels of information technology understanding.

• At a high level, an organization should maintain an end-to-end diagram that depicts the core elements of its system, such as source systems that collect data, databases, and sanctions screening or AML (“target”) applications. Additional documentation and diagrams should provide further details on data flow from “source” to “target” data, and should include the following:
  • Data dictionaries: Descriptions of the contents, format, and structure of fields within source and target tables; and
  • Data lineage documents: Information describing which data elements (particularly for mission critical data points, such as those used in transaction monitoring) are passed from source systems into target systems, explaining any transformations or changes to the data elements that take place.

This collection of documents, which provides end-to-end details of the data and system infrastructure, should be updated as different components of the system change, and periodically reviewed to confirm accuracy.

Why is this considered a leading practice? This documentation provides detailed information about the system architecture and permits a third party, such as an auditor or regulator, to understand how the system works and facilitates dialogue should further analysis or testing be required.

Leading Practice #2: Evidence of Testing That Is Clear and Detailed

Financial institutions generally understand the need for robust testing to be performed so that they can use their applications with confidence. However, the documentation of such testing varies widely. The Guidance is clear on what is to be expected: “Included in testing activities should be the purpose, design, and execution of test plans, summary results with commentary and evaluation, and detailed analysis of informative samples. Testing activities should be appropriately documented.”⁴ Many financial institutions have not developed or maintained supporting documentation that comprehensively describes the testing conducted, or solely describe samples that were drawn at the described level of detail.

Below are some leading practices that can help an organization detail key components of the testing process:

1. Consider the documentation of testing to be done explicitly for the benefit of a third party; testing itself is no longer done simply to establish that systems work, but to demonstrate to others that they work.
2. Describe the organization’s methodology in determining an appropriate sample when samples are drawn. The documentation should also include data on the sample that was used for testing, and how it is reliable for the purposes of drawing conclusions about the population.
3. Indicate both the results of testing and how the results should be interpreted. For example, if the test failed, documentation should describe what the issue is, if known, and how the issue will be resolved.

Why is this considered a leading practice? Data testing demonstrates that data is accurate and complete, and testing (or the review of historical testing) is a necessary component of any validation effort. The clearer and more informative testing is, the easier it will be to demonstrate adherence to the Guidance.

Leading Practice #3: Comprehensive ETL Controls

Customer and transactional data within financial organizations is often complex. This data is extracted, transformed and loaded—potentially multiple times—from source systems to target applications. As a result, there are multiple opportunities where data may be lost or processes may fail. Financial institutions are expected to develop comprehensive (and automated) ETL controls. The Guidance states: “Sound model risk management depends on substantial investment in supporting systems to ensure data and reporting integrity, together with controls and testing to ensure proper implementation of models, effective systems integration, and appropriate use.”⁵ Without controls, organizations may fail to know when issues have occurred within the ETL process. We have identified several leading practices for controls that financial institutions can consider implementing.

1. Develop an ETL execution control each time data is processed to another system, database or application. For example, a log or automated email confirming successful/unsuccessful execution of a job, including the count of successful records transmitted, is one such control.
2. Develop rigorous escalation procedures in case there are errors with ETL processes, such as an email to the vendor, notifying them of a corrupt file; include expected time to resolve certain steps (e.g., parties are expected to respond within 24 hours or a further escalation will occur).

Why is this considered a leading practice? Compliance applications are mission-critical, and errors that compromise the data for these applications, especially when those errors are not identified in a timely manner, can raise questions about the reliability of these applications. Demonstrating a robust set of controls can inspire confidence in the ETL architecture.

As financial institutions prepare for model validations, they should consider these three leading practices for DC&I. By developing an understanding of data and system architecture, documenting evidence of testing conducted and implementing ETL controls, financial institutions can be better-positioned to demonstrate that they are in alignment with the Guidance.

Donald Monson, senior manager, Deloitte Risk and Financial Advisory, Deloitte Transactions and Business Analytics LLP, Chicago, IL, dmonson@deloitte.com

Albert Hong, manager, Deloitte Risk and Financial Advisory, Deloitte Transactions and Business Analytics LLP, McLean, VA, albhong@deloitte.com

1. “Supervisory Guidance on Model Risk Management,” Office of the Comptroller of the Currency, April 4, 2011, https://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf
2. “SR 11-17: Guidance on Model Risk Management,” Board of Governors of the Federal Reserve System, https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
3. “Supervisory Guidance on Model Risk Management,” Office of the Comptroller of the Currency, April 4, 2011, https://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf
4. Ibid.
5. Ibid.