ISACA Now Blog

Knowledge & Insights > ISACA Now > Posts > Empower Auditors to Think Big Picture on AI

Empower Auditors to Think Big Picture on AI

Andrew Clark, Data Economist, Block Science
| Posted at 3:05 PM by ISACA News | Category: Audit-Assurance | Permalink | Email this Post | Comments (1)

Andrew ClarkThe new white paper, Auditing Artificial Intelligence, provides an overview of what AI is, why auditors need to be aware of AI, and how the COBIT 2019 framework relates to AI auditing.

The guidance addresses the somewhat nebulous definition of AI, as there is no agreed-upon definition even in the research community, since AI encompasses a wide swath of ground, including machine learning, deep learning (a subset of machine learning), and some types of rule-based systems. ISACA wisely takes a neutral stance regarding definitions and capabilities of AI, as fact vs. fiction is still under active investigation.

As AI implementations are still in embryonic stages of deployment for the vast majority of companies outside of Silicon Valley, and there is a lack of regulatory requirements for assuring AI, there is still no definitive and comprehensive set of auditing standards for AI. Research is progressing, however—with ISACA at the forefront, as this whitepaper and its cited papers suggest.

The shortage of specialized tech talent for implementations, the “black box” nature of AI, and the deficiency of research regarding the holistic impact of AI on organizations, are just some of the challenges confronting IT auditors who are tasked with auditing AI. Approaches for addressing the black box nature of algorithms exist, such as sensitivity analysis and the like, but these approaches are often time-consuming and best left to modeling specialists for technical evaluation. The paper makes a recommendation for bifurcating an audit of AI between model specialists and IT auditors, with IT auditors looking at the holistic process and how technology stacks integrate. The authors highlight that in small-to-medium-sized enterprises that implement AI, third-party vendor management may be one of the critical aspects of an audit. The use of vendors allows less technical users to access the AI solution; however, keep in mind that many vendor products cannot be customized.

The paper states that auditors look at the holistic risks and integration of AI into the organization, and approach AI as they have approached cybersecurity and cloud computing, with an iterative, adaptive approach focusing on the implications. Commonly, auditors mistakenly believe that they need to know the low-level details of how algorithms work before conducting an AI audit. This is not the case, and it may actually be more beneficial when auditors do not know the intricate details of how AI works, as they will be able to take a holistic, 40,000-foot view into how AI makes sense in the enterprise instead of getting caught in the weeds.

I believe that this is the area that is currently missing the most in enterprises: a holistic view of AI. Technologies now rule the roost, but no matter how impressive the technical capabilities, an AI system needs to make sense in the grand mission of the company. The technological sophistication takes a backseat, and sometimes a less technical system that is more controllable is better for organizations. Let the technologists take care of the technical details, and empower auditors to think big picture, which is where they can provide tremendous value and shine.

The paper concludes with context of how COBIT 2019 can be used to create an audit plan for AI, along with an enumeration of the nine main challenges to an effective AI audit that ISACA has identified, with similar best practice approaches to tackle these challenges.

Auditing Artificial Intelligence is arguably the most comprehensive analysis of the current state of AI auditing, governance and assurance. It is the ideal stepping-off point for beginning your governance analysis and planning an AI audit for your enterprise.

Copy Item to All Language Codes
Lists/SqtResources/AllItems.aspx
0x0
0x0
ContentType
0x01009AF1BC4E56474a80B49512D1B30D6EEC
225
Manage Subscriptions
/_layouts/images/ReportServer/Manage_Subscription.gif
/Knowledge-Center/Blog/_layouts/ReportServer/ManageSubscriptions.aspx?list={ListId}&ID={ItemId}
0x80
0x0
FileType
rdl
350
Manage Data Sources
/Knowledge-Center/Blog/_layouts/ReportServer/DataSourceList.aspx?list={ListId}&ID={ItemId}
0x0
0x20
FileType
rdl
351
Manage Parameters
/Knowledge-Center/Blog/_layouts/ReportServer/ParameterList.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rdl
352
Manage Processing Options
/Knowledge-Center/Blog/_layouts/ReportServer/ReportExecution.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rdl
353
View Report History
/Knowledge-Center/Blog/_layouts/ReportServer/ReportHistory.aspx?list={ListId}&ID={ItemId}
0x0
0x40
FileType
rdl
354
View Dependent Items
/Knowledge-Center/Blog/_layouts/ReportServer/DependentItems.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rsds
350
Edit Data Source Definition
/Knowledge-Center/Blog/_layouts/ReportServer/SharedDataSource.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rsds
351
View Dependent Items
/Knowledge-Center/Blog/_layouts/ReportServer/DependentItems.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
smdl
350
Manage Tapthrough Reports
/Knowledge-Center/Blog/_layouts/ReportServer/ModelTapThrough.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
smdl
352
Manage Model Item Security
/Knowledge-Center/Blog/_layouts/ReportServer/ModelItemSecurity.aspx?list={ListId}&ID={ItemId}
0x0
0x2000000
FileType
smdl
353
Regenerate Model
/Knowledge-Center/Blog/_layouts/ReportServer/GenerateModel.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
smdl
354
Manage Data Sources
/Knowledge-Center/Blog/_layouts/ReportServer/DataSourceList.aspx?list={ListId}&ID={ItemId}
0x0
0x20
FileType
smdl
351
Load in Report Builder
/Knowledge-Center/Blog/_layouts/ReportServer/RSAction.aspx?RSAction=ReportBuilderModelContext&list={ListId}&ID={ItemId}
0x0
0x2
FileType
smdl
250
Edit in Report Builder
/_layouts/images/ReportServer/EditReport.gif
/Knowledge-Center/Blog/_layouts/ReportServer/RSAction.aspx?RSAction=ReportBuilderReportContext&list={ListId}&ID={ItemId}
0x0
0x4
FileType
rdl
250
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XsnLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
FileType
xsn
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document.2
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document.3
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document.4
255
View in Web Browser
/_layouts/images/ichtmxls.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&DefaultItemOpen=1
0x0
0x1
FileType
xlsx
255
View in Web Browser
/_layouts/images/ichtmxls.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&DefaultItemOpen=1
0x0
0x1
FileType
xlsb
255
Snapshot in Excel
/_layouts/images/ewr134.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&Snapshot=1
0x0
0x1
FileType
xlsx
256
Snapshot in Excel
/_layouts/images/ewr134.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&Snapshot=1
0x0
0x1
FileType
xlsb
256

Comments

ISACA Auditors for the 21st century and beyond

This is a sound approach to any new tech and in particularly the rounded approach that ISACA auditors take, linking back to the org strategy and governance that is critical in establishing how AI is used and that it is being used ethically, those people who need to know how it works and that it provides VFM and delivers to expectations, understand key risks and opportunities at senior and operational level.  This also shows how Cobit 5 and now CObit 2019 provides a flexible framework to direct audits and help auditors to focus on key risks and to provide correct assurance to Audit Committee.  Like stated there is lots more to learn and in particularly for me there is a lot of ethical issues to understand in using the technologies and ensuring that this matches the companies is key as is ensuring that they comply with legislation in that country and regulations such as H&S etc which may be not apparent of fully understand stood until later. 
Paul Ridley at 1/14/2019 6:07 AM
Copy Item to All Language Codes
Lists/SqtResources/AllItems.aspx
0x0
0x0
ContentType
0x01009AF1BC4E56474a80B49512D1B30D6EEC
225
Manage Subscriptions
/_layouts/images/ReportServer/Manage_Subscription.gif
/Knowledge-Center/Blog/_layouts/ReportServer/ManageSubscriptions.aspx?list={ListId}&ID={ItemId}
0x80
0x0
FileType
rdl
350
Manage Data Sources
/Knowledge-Center/Blog/_layouts/ReportServer/DataSourceList.aspx?list={ListId}&ID={ItemId}
0x0
0x20
FileType
rdl
351
Manage Parameters
/Knowledge-Center/Blog/_layouts/ReportServer/ParameterList.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rdl
352
Manage Processing Options
/Knowledge-Center/Blog/_layouts/ReportServer/ReportExecution.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rdl
353
View Report History
/Knowledge-Center/Blog/_layouts/ReportServer/ReportHistory.aspx?list={ListId}&ID={ItemId}
0x0
0x40
FileType
rdl
354
View Dependent Items
/Knowledge-Center/Blog/_layouts/ReportServer/DependentItems.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rsds
350
Edit Data Source Definition
/Knowledge-Center/Blog/_layouts/ReportServer/SharedDataSource.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
rsds
351
View Dependent Items
/Knowledge-Center/Blog/_layouts/ReportServer/DependentItems.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
smdl
350
Manage Tapthrough Reports
/Knowledge-Center/Blog/_layouts/ReportServer/ModelTapThrough.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
smdl
352
Manage Model Item Security
/Knowledge-Center/Blog/_layouts/ReportServer/ModelItemSecurity.aspx?list={ListId}&ID={ItemId}
0x0
0x2000000
FileType
smdl
353
Regenerate Model
/Knowledge-Center/Blog/_layouts/ReportServer/GenerateModel.aspx?list={ListId}&ID={ItemId}
0x0
0x4
FileType
smdl
354
Manage Data Sources
/Knowledge-Center/Blog/_layouts/ReportServer/DataSourceList.aspx?list={ListId}&ID={ItemId}
0x0
0x20
FileType
smdl
351
Load in Report Builder
/Knowledge-Center/Blog/_layouts/ReportServer/RSAction.aspx?RSAction=ReportBuilderModelContext&list={ListId}&ID={ItemId}
0x0
0x2
FileType
smdl
250
Edit in Report Builder
/_layouts/images/ReportServer/EditReport.gif
/Knowledge-Center/Blog/_layouts/ReportServer/RSAction.aspx?RSAction=ReportBuilderReportContext&list={ListId}&ID={ItemId}
0x0
0x4
FileType
rdl
250
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XsnLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
FileType
xsn
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document.2
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document.3
255
Edit in Browser
/_layouts/images/icxddoc.gif
/Knowledge-Center/Blog/_layouts/formserver.aspx?XmlLocation={ItemUrl}&OpenIn=Browser
0x0
0x1
ProgId
InfoPath.Document.4
255
View in Web Browser
/_layouts/images/ichtmxls.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&DefaultItemOpen=1
0x0
0x1
FileType
xlsx
255
View in Web Browser
/_layouts/images/ichtmxls.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&DefaultItemOpen=1
0x0
0x1
FileType
xlsb
255
Snapshot in Excel
/_layouts/images/ewr134.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&Snapshot=1
0x0
0x1
FileType
xlsx
256
Snapshot in Excel
/_layouts/images/ewr134.gif
/Knowledge-Center/Blog/_layouts/xlviewer.aspx?listguid={ListId}&itemid={ItemId}&Snapshot=1
0x0
0x1
FileType
xlsb
256
You must be logged in and a member to post a comment to this blog.