Get started with Azure AD Report Only Mode for Conditional Access

The Microsoft Azure AD Team has just released a long awaited feature in public preview. That feature is called Azure AD Report Only Mode for Conditional Access. Report-only mode allows administrators to evaluate the impact of Conditional Access policies before enabling them in their environment.

What is it?

While working with Conditional Access you might have realized that even though you have made all the plans in the world, somethings did not pan out quite as expected. So you turn to the “What If”-tool in Conditional Access to see what is going on. A great tool, but only after the fact that your user already are locked out of his application. This is because the What If tool only looks at the impact of your enabled policies with it’s current targeting. Think of this as a tool to ONLY see the current impact of your existing policies and targeting. Now with Report Only mode we will be able to create a new Conditional Access Policy, target the services, apps, locations and all we want and set it in Audit mode. This means that this policies will NOT apply to the users sessions, but it will log and report on the result that would have been if the policy was applied. Report Only mode allows us to look back a few days and see how our new policy would have affected our users BEFORE we active it. Great? I think so.

Base setup for optimal insight

Report only mode gives us 2 different ways to look at the results where the Workbook is the way that gives us the best insight. To be able to use the workbook, we do need a Azure Log Analytics Workspace where we forward the Azure AD logs too. The other way that does not require log analytics is to look directly in the Azure AD Signin logs and go through each and every logon to see the effect. Not very effective indeed. Both views have their purpose that I will show you soon. Let us start with some recommended requirements:

  • An Azure Subscription
  • A log analytics workspace
  • Azure AD Premium P1 or higher (to be able to export the logs)
  • Azure AD Log Diagnostics configured to forward the logs to our Log Analytics Workspace.

How to use this to test and verify your new policy setup

First step is to setup Azure Monitor integration with Azure AD if you don’t already have it. I will not go into the details of the setup of that in this post. So assuming you have forwarded your Azure AD Logs to a Log analytics Workspace you can go ahead and create a conditional access policy. So make whatever number of polices you want you want to assess, but at the stage where you normally choose to activate the policy you instead choose “Report-Only”.

Now you wait. Let the policies be in this state for some time (maybe a week or 2)

NOTE: It you create policies that require compliant devices for users on Mac, iOS and Android. They may be prompted to select a device certificate during policy evaluation, even though device compliance is not enforced. Educate your users before you turn that on, or maybe not include Mac, iOS or Android in your targets during testing. But if you exclude them, you will not get a full picture of the impact of your policies.

After a week or so we are ready to really have a look back on the impact of our policy setup. For that we are going to use the Conditional Access Insights workbook. Go to Azure AD in your portal and under Monitoring you will find the Workbook blade.

Choose the Conditional Access Insights workbook. The start view in this workbook is your current ACTIVATED policies. To be able to assess your report-only policies you will need to change your view.

What we see here is that I have 3 report-only policies setup and none of them are selected. If I want to see the combined result of all of them, I simply select them all. If I want too see the impact of removing one of my active policies and replace it with a new one, I need to deselect my active policy and select the one I am replacing it with like this:

My view will not change to reflect that TestPolicy01 is active and that my All Users: Block Lecagy Authentication is disabled.

Impact Summary

User View
s View

The Impact Summary shows us the combined result of all policies we have selected. It can be in either User view of Sign-ins view. We have 4 different results:

  • Success: All selected policies where evaluated to be true and the user where able to logon to his application. For example the requirement is a Hybrid AD Joined device, and the user is meeting that criteria.
  • Failure: A selected policy failed. This means the users would have been blocked from logon because the requirements was not met. For example a compliant device is required, and the user is not coming from a compliant device and therefore would be blocked.
  • User Action Required: A selected policy applied and user action (like MFA) would have been required. This result is only applicable for report-only polices. The user is not prompted to satisfy the required control.
  • Not Applied: A logon was made where none of your selected policies applied. One example is that the user is excluded from the policy

This report is a great view for seeing the all up view on your organization and you can drill down in the workbook or jump over to Log Analytics to query the data directly as well. You also will see this data in the User Signin logs. When you find a user having multiple failed logons , go to Azure AD to look at the sign-in logs for that user.

The Sign-in Logs view

If you go into a users specific sign-in log you when you click on a sign-in you get the details pane below.

So in this view we now have a new choice called Report-Only (Preview) option. When we click on that we will see exactly how the different policies would be applied to this specific logon.

As we can see here 2 off my current test-policies would have been blocking this users from logon, while one of my policies would not have been applied to this specific logon.

Remember this is still in preview. But this is a most welcome addition. Having the opportunity the look backwards an how your policies would have impacted your users if you enabled them is truly insightful and allows us to have much better control when doing changes to our conditional access setup.

The official Microsoft documentation can be found here:

Jan Ketil Skanke

Jan Ketil is an Enterprise Mobility MVP since 2016 and are working as a COO and Principal Cloud Architect at CloudWay in Norway. He has been in the industry for more than 20 years working for both Microsoft Partners and Microsoft. He loves to speak about anything around Enterprise Mobility and Secure Productivity. He is also the lead for the community conference Experts Live Norway. Jan Ketil has presented at large industry conferences like Microsoft Ignite, Microsoft Ignite The Tour, Microsoft Inspire, Experts Live Europe, Techmentor HQ (3rd best session 2019) and NIC Conference in Oslo.

Add comment


Categories use cookies to ensure that we give you the best experience on our website.