Troubleshooting connectivity with Network Watcher

Scenario

You’re troubleshooting an issue where vm‑app cannot connect to vm‑db on port 3306 (MySQL). Both VMs live in different VNets, each with its own NSG, and the environment includes VNET peering. Something in the network path, an NSG rule, routing issue, or filtering policy is blocking traffic, but it’s not immediately clear where the failure is happening.

Azure Network Watcher’s Connection Troubleshoot tool helps you trace the path, identify whether the traffic is allowed or denied, and pinpoint the exact component causing the connectivity failure.

Lab Objectives

• Use Azure Network Watcher to diagnose connectivity between two VMs
• Run Connection Troubleshoot to test port‑level communication
• Identify whether NSGs, routing, or peering are blocking traffic
• Validate outbound and inbound rules using Network Watcher insights
• Confirm the fix by re‑running the connectivity test

Prerequisites

• Azure subscription
• Network Watcher enabled in the region (Azure enables it automatically when a VNET is created)
• Two VMs deployed in separate VNets (vm‑app and vm‑db)
• NSGs applied to NICs or subnets
• VNET peering configured between the VNets

The Lab

We’ll use the following setup:

^*Use the following commands to install Apache web server on vm-app

sudo apt update && sudo apt install -y apache2 && echo -e "Listen 0.0.0.0:80\nListen [::]:80" | sudo tee /etc/apache2/ports.conf > /dev/null && sudo systemctl restart apache2

⁺Use the following commands to install MySQL database server on vm-db

sudo apt update && sudo apt install -y mysql-server && sudo sed -i "s/^bind-address.*/bind-address = 0.0.0.0/" /etc/mysql/mysql.conf.d/mysqld.cnf && sudo systemctl restart mysql

The Why

NSGs are often the first place to look when traffic between Azure resources isn’t flowing. But with rules applied at both subnet and NIC levels, it’s easy to miss something. Azure Network Watcher provides tools to simulate traffic, inspect effective rules, and pinpoint the exact rule causing a block.

Troubleshooting using Network Watcher

Network Watcher automatically creates a resource group named NetworkWatcherRG that holds any resources used by Network Watcher tools:

• IP Flow Verify Checks if NSG rules allow or deny specific traffic between a source and destination IP, port, and protocol. Useful for pinpointing NSG rule conflicts.
• NSG Diagnostics Provides insights into NSG configuration, including rule conflicts, priority issues, and traffic flow analysis. Helps validate NSG setup across NICs and subnets.
• Next Hop Determines the next hop for a packet from a VM to a destination IP. Useful for verifying routing paths and peering configurations.
• Effective Security Rules Displays the cumulative NSG rules applied to a VM’s NIC or subnet. Helps understand which rules are actually enforced on the VM.
• Connection Troubleshoot Simulates traffic between endpoints to test connectivity. Evaluates routing, NSGs, and listener availability, making it ideal for end-to-end diagnostics.

We’ll use these tools to troubleshoot the misconfigurations created in this lab.

Test 1 - IP Flow Verify

To test TCP connectivity from vm-app to vm-db on port 3306 using source port 54321 (just a random high-numbered port like a real app might use). The test failed due to a deny rule for VNetOutbound in nsg-app.

• Source VM: vm-app
• Destination IP: 10.2.0.4 (IP of vm-db)
• Destination Port: 3306 (MySQL)
• Source Port: 54321 (just a random high-numbered port like a real app might use)
• Result: Access Denied
• Reason: Deny rule for VNetOutbound in nsg-app blocked the traffic

To test TCP connectivity from vm-db to vm-app on port 80 using source port 54321 (just a random high-numbered port like a real app might use). The test failed due to the default deny rule in nsg-db.

• Source VM: vm-db
• Destination IP: 10.1.0.4 (IP of vm-app)
• Destination Port: 80 (HTTP)
• Source Port: 54321 (just a random high-numbered port like a real app might use)
• Result: Access Denied
• Reason: Default deny rule in nsg-db blocked the traffic

Test 2 - NSG Diagnostics

To analyse traffic flow from vm-app to vm-db on port 3306. The test confirmed that the traffic was denied due to a rule applied on the NIC of vm-app.

• Target VM: vm-app
• Protocol: TCP
• Direction: Outbound
• Source Type: IPv4 address/CIDR
• IPv4 address/CIDR: 10.1.0.4 (IP of vm-app)
• Destination IP: 10.2.0.4 (IP of vm-db)
• Destination Port: 3306 (MySQL)
• Result: Access Denied
• Reason: NSG rule on vm-app NIC blocked outbound traffic to vm-db

This test evaluated outbound traffic from vm-db to vm-app on port 80. The traffic was denied due to an NSG rule applied at the subnet level.

• Target VM: vm-db
• Protocol: TCP
• Direction: Outbound
• Source Type: IPv4 address/CIDR
• Source IP: 10.2.0.4 (IP of vm-db)
• Destination IP: 10.1.0.4 (IP of vm-app)
• Destination Port: 80 (HTTP)
• Result: Access Denied
• Reason: NSG rule on subnet-db blocked outbound traffic to vm-app

Test 3 – Next Hop

This test tells us where a network packet will next be routed from source in process to get to its destination.

vm-app to vm-db

• VM: vm-app
• Source IP: 10.1.0.4 (IP of vm-app)
• Destination IP: 10.2.0.4 (IP of vm-db)
• Result: Next hop is VirtualNetworkPeering, which gives us a clue that we should check VNet peering settings.

vm-db to vm-app

• VM: vm-db
• Source IP: 10.2.0.4 (IP of vm-db)
• Destination IP: 10.1.0.4 (IP of vm-app)
• Result: Next hop is VirtualNetworkPeering, which gives us a clue that we should check VNet peering settings.

Test 4 – Effective Security Rules

This test examined the effective NSG rules applied to each VM to understand how traffic is being filtered at both the NIC and subnet levels.

vm-app

• NSG Location: Applied directly to the NIC.
• Effective Rules: Reflect only the NSG associated with the NIC.
• Implication: Traffic filtering is controlled at the NIC level.

vm-db

• NSG Location: Applied to db-subnet
• Effective Rules: Reflect NSG rules inherited from the subnet.
• Implication: All VMs within the subnet share the same NSG rules.

Test 5 - Connection Troubleshoot Tool

This test simulates full connectivity diagnostics between two virtual machines. It effectively generates a report of the previous tests we ran. The following settings were used:

vm-app to vm-db

• Source Type: Virtual machine
• Source VM: vm-app
• Destination Type: Virtual machine
• Destination VM: vm-db
• Preferred IP Version: IPv4
• Protocol: TCP
• Destination Port: 3306
• Source Port: 54321 (just a random high-numbered port like a real app might use)
• Diagnostic Tests: Connectivity, NSG diagnostic, Next hop, Port scanner

vm-db to vm-app

• Source Type: Virtual machine
• Source VM: vm-db
• Destination Type: Virtual machine
• Destination VM: vm-app
• Preferred IP Version: IPv4
• Protocol: TCP
• Destination Port: 80
• Source Port: 54321 (just a random high-numbered port like a real app might use)
• Diagnostic Tests: Connectivity, NSG diagnostic, Next hop, Port scanner

Conclusion

Azure Network Watcher makes connectivity troubleshooting far more precise. Instead of guessing which rule is the culprit, you get a clear answer fast. Whether you're debugging a blocked port or validating your security posture, these tools are essential for any Azure engineer.

Lab Video