Active/Standby Data Centre Network Design using GNS3/Virtualbox/JUNOS and Cisco – Part-5

In this tutorial I am going to implement IPSEC VPN between karachigw1/lahoregw1 and cloud firewalls karacf3/lahorcf3.

Final Topology

  1. Configure karachigw1 as following:

Define Phase1 & 2 and set both karacf3/larhocf3 as peers

!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
crypto isakmp key cisco123 address 10.236.7.171
crypto isakmp key cisco123 address 10.236.7.179
!
!
crypto ipsec transform-set ptcltrans esp-3des esp-md5-hmac 
 mode transport
!
!
crypto map ptclmap local-address Loopback0
crypto map ptclmap 10 ipsec-isakmp 
 set peer 10.236.7.171
 set peer 10.236.7.179
 set transform-set ptcltrans 
 match address 100
!

Apply profile on interface:

!
interface FastEthernet0/0.116
 description to_karais4
 encapsulation dot1Q 116
 ip address 18.18.18.2 255.255.255.248
 crypto map ptclmap
!

Define encryption domain

!
access-list 100 permit ip 141.97.0.0 0.0.255.255 10.144.58.0 0.0.0.255
access-list 100 permit ip 192.168.1.0 0.0.0.255 10.144.58.0 0.0.0.255
!

Create loopback Interface

!
interface Loopback0
 ip address 20.20.20.1 255.255.255.255
!

Define route-map to advertise the loopback with no prepend

!
!
ip access-list standard floating-loopback
 permit 20.20.20.1
!
route-map no-prepend permit 10
 match ip address floating-loopback
!

Apply the route-map to the BGP peer

!
router bgp 65120
 neighbor 18.18.18.1 route-map no-prepend out
 no auto-summary
!
  1. Configure lahoregw1 as following:

Define Phase1 & 2 and set both karacf3/larhocf3 as peers

!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
crypto isakmp key cisco123 address 10.236.7.171
crypto isakmp key cisco123 address 10.236.7.179
!
!
crypto ipsec transform-set ptcltrans esp-3des esp-md5-hmac 
 mode transport
!
crypto map ptclmap local-address Loopback0
crypto map ptclmap 10 ipsec-isakmp 
 set peer 10.236.7.171
 set peer 10.236.7.179
 set transform-set ptcltrans 
 match address 100
!

Apply crypto map on interface

!
interface FastEthernet0/0.117
 description to_lahoris4
 encapsulation dot1Q 117
 ip address 19.19.19.2 255.255.255.248
 crypto map ptclmap
!
!

Define encryption domain

!
access-list 100 permit ip 141.97.0.0 0.0.255.255 10.144.58.0 0.0.0.255
access-list 100 permit ip 192.168.1.0 0.0.0.255 10.144.58.0 0.0.0.255
!

Create loopback Interface

!
interface Loopback0
 ip address 20.20.20.1 255.255.255.255
!
!

Define route-map to pre-pend the loopback advertisement so that this not preferred.

!
ip access-list standard floating-loopback
 permit 20.20.20.1
!
!
route-map prepend permit 10
 match ip address floating-loopback
 set as-path prepend 65120 65120 65120
!

Apply route-map to BGP peer

!
router bgp 65120
 neighbor 19.19.19.1 route-map prepend out
 no auto-summary
!

Now at the other side, configure IPSEC VPN on both Cisco ASAv primary cloud firewall karacf3 and DR firewall lahorcf3 as following:

crypto ipsec ikev1 transform-set ptcltransform esp-3des esp-md5-hmac 
crypto ipsec security-association pmtu-aging infinite
crypto map ptclmap 1 match address ptcl-vpn-lan
crypto map ptclmap 1 set peer 20.20.20.1 
crypto map ptclmap 1 set ikev1 transform-set ptcltransform
crypto map ptclmap interface on-ramp
crypto ca trustpool policy
crypto ikev1 enable on-ramp
crypto ikev1 policy 1
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
!
tunnel-group 20.20.20.1 type ipsec-l2l
tunnel-group 20.20.20.1 ipsec-attributes
 ikev1 pre-shared-key cisco123
!
access-list ptcl-vpn-lan extended permit ip 10.144.58.0 255.255.255.0 141.97.0.0 255.255.0.0 
access-list ptcl-vpn-lan extended permit ip 10.144.58.0 255.255.255.0 192.168.1.0 255.255.255.0 
!

Now configure test machine Service-vlan102 in dc-karachi with IP address 10.144.58.2/24 and connect it to local aggregation switch karais3. Similarly configure test machine Service-vlan103 in dc-lahore with IP address 10.144.58.2/24.

In customer’s offices, configure a LAN user PC1 in Karachi Office with IP address 141.97.1.10/16 and PC2 with IP address 192.168.1.10/24 in Lahore office. Connect these two test PC to the Layer-3 switch ESW1 and ESW2.

Configure ESW1 as following:

!
interface FastEthernet1/15
 switchport access vlan 104
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan104
 ip address 141.97.1.1 255.255.0.0
!
router eigrp 100
 network 141.97.0.0
 network 172.168.0.0
 auto-summary
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 172.168.3.1
!

Configure ESW2 as following:

!
interface Vlan104
 ip address 192.168.1.1 255.255.255.0
!
router eigrp 100
 network 172.168.0.0
 network 192.168.1.0
 auto-summary
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 172.168.4.1
!

Both ESW1 and ESW2 will forward their traffic to R1 and R2 respectively. Configuration of R1 and R2 is following:

R1:

!
interface FastEthernet0/0
 ip address 172.168.2.1 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet0/1
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface FastEthernet1/0
 ip address 172.168.3.1 255.255.255.0
 duplex auto
 speed auto
!

Apply HSRP configuration on this interface.

!
interface FastEthernet2/0
 ip address 172.168.1.5 255.255.255.240
 duplex auto
 speed auto
 standby 1 ip 172.168.1.4
 standby 1 timers 1 3
 standby 1 priority 120
 standby 1 preempt
!
router eigrp 100
 network 172.168.0.0
 auto-summary
!

Set default route as following that will be HSRP VIP address of karachigw1 and lahoregw2 which we will configure later:

!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 172.168.1.1
!

Similarly configure R2 as following:

!
interface FastEthernet0/0
 ip address 172.168.2.2 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet0/1
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface FastEthernet1/0
 ip address 172.168.4.1 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet2/0
 ip address 172.168.1.6 255.255.255.240
 duplex auto
 speed auto
 standby 0 preempt
 standby 1 ip 172.168.1.4
 standby 1 timers 1 3
!
router eigrp 100
 network 172.168.0.0
 auto-summary
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 172.168.1.1
!

Now configure karachigw1 as following.

  1. Apply IP SLA configuration as following:
!
ip sla 1
 icmp-echo 10.236.7.171 source-interface FastEthernet0/0.116
 timeout 1000
 threshold 1000
 frequency 3
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo 172.168.1.4 source-interface FastEthernet1/0
 timeout 1000
 threshold 1000
 frequency 3
ip sla schedule 2 life forever start-time now
ip sla 3
 icmp-echo 18.18.18.1 source-interface FastEthernet0/0.116
 timeout 1000
 threshold 1000
 frequency 1
ip sla schedule 3 life forever start-time now
!
!         
track 1 ip sla 1 reachability
 delay down 10 up 10
!
track 2 ip sla 2 reachability
 delay down 10 up 10
!
track 3 list boolean and
 object 1
 object 2
!
track 4 ip sla 3 reachability
!
track 5 ip route 10.236.7.168 255.255.255.248 reachability
!
  1. Apply HSRP and Object tracking configuration as following:
!
interface FastEthernet1/0
 ip address 172.168.1.2 255.255.255.240
 duplex auto
 speed auto
 standby 0 ip 172.168.1.1
 standby 0 timers 1 3
 standby 0 priority 120
 standby 0 preempt
 standby 0 track 3 decrement 30
!
  1. Configure static routing as following:
!
ip route 18.18.18.0 255.255.255.248 Null0 track 4
ip route 141.97.0.0 255.255.0.0 172.168.1.4
ip route 192.168.1.0 255.255.255.0 172.168.1.4

Similarly; Apply configuration on lahoregw1 as following:

  1. Apply IP SLA configuration as following:
!
ip sla 1
 icmp-echo 10.236.7.179 source-interface FastEthernet0/0.117
 timeout 1000
 threshold 1000
 frequency 3
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo 172.168.1.4 source-interface FastEthernet1/0
 timeout 1000
 threshold 1000
 frequency 3
ip sla schedule 2 life forever start-time now
ip sla 3
 icmp-echo 19.19.19.1 source-interface FastEthernet0/0.117
 timeout 1000
 threshold 1000
 frequency 1
ip sla schedule 3 life forever start-time now
!
!         
track 1 ip sla 1 reachability
 delay down 10 up 10
!
track 2 ip sla 2 reachability
 delay down 10 up 10
!
track 3 list boolean and
 object 1
 object 2
!
track 4 ip sla 3 reachability
!
track 5 ip route 10.236.7.176 255.255.255.248 reachability
!
  1. Apply HSRP and Object tracking configuration as following:
!
interface FastEthernet1/0
 ip address 172.168.1.3 255.255.255.240
 duplex auto
 speed auto
 standby 0 ip 172.168.1.1
 standby 0 timers 1 3
 standby 0 preempt
 standby 0 track 3 decrement 30
end
  1. Re-configure the static routing as following:
!
ip route 19.19.19.0 255.255.255.248 Null0 track 4
ip route 141.97.0.0 255.255.0.0 172.168.1.4
ip route 192.168.1.0 255.255.255.0 172.168.1.4

Now let’s check the status of karachigw1, lahoregw1 and R1, R2 to identify which one is our Active HSRP router:

karachigw1#sh standby brief 
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa1/0       0    120 P Active  local           172.168.1.3     172.168.1.1

karachigw1#
lahoregw1#sh standby brief 
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa1/0       0    100 P Standby 172.168.1.2     local           172.168.1.1
lahoregw1#

Similarly; Check on R1 and R2:

R1#sh standby brief 
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa2/0       1    120 P Active  local           172.168.1.6     172.168.1.4
R1#
R2#sh standby brief 
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa2/0       1    100   Standby 172.168.1.5     local           172.168.1.4
R2#

Let’s try to ping from PC1 141.97.1.10 to the gateway VIP address 172.168.1.1:

Checking for duplicate address...
PC1 : 141.97.1.10 255.255.0.0 gateway 141.97.1.1
PC1> 
PC1> 
PC1> ping 172.168.1.1
172.168.1.1 icmp_seq=1 timeout
84 bytes from 172.168.1.1 icmp_seq=2 ttl=253 time=31.650 ms
84 bytes from 172.168.1.1 icmp_seq=3 ttl=253 time=36.967 ms
84 bytes from 172.168.1.1 icmp_seq=4 ttl=253 time=26.586 ms
84 bytes from 172.168.1.1 icmp_seq=5 ttl=253 time=29.608 ms
PC1>

Let’s check on PE routers karair3 and lahorir3 that from where we are learning the remote encryption domain network 10.144.58.0/24. Please note this is being advertised from both Cloud firewalls. However this is an internal network and customer will not be advertised to the customer’s gateway routers karachigw1 and larhoregw1.

Note that; we are learning 10.144.58.0/24 from karacf3 firewall on PE karair3:

root@karair3> show route table vpn12725.inet.0 
vpn12725.inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.144.58.0/24     *[BGP/170] 00:45:55, MED 0, localpref 100
                      AS path: 65119 ?
                    > to 10.236.7.171 via em3.112
10.236.7.168/29    *[Direct/4] 00:48:49
                    > via em3.112
10.236.7.176/29    *[BGP/170] 00:48:14, localpref 100, from 2.2.2.2
                      AS path: I
                    > to 40.50.60.2 via em0.0, Push 18, Push 299776(top)
18.18.18.0/29      *[Direct/0] 00:48:50
                    > via em1.116
18.18.18.1/32      *[Local/0] 00:48:50
                      Local via em1.116
20.20.20.1/32      *[BGP/170] 00:48:10, MED 0, localpref 100
                      AS path: 65120 I
                    > to 18.18.18.2 via em1.116
root@karair3>

Similarly please note that; we are learning 10.144.58.0/24 from both karacf3 and lahorcf3 firewall on PE lahorir3 but only route learned from karacf3 is being installed in the routing table:

root@lahoreir3> show route table vpn12725.inet.0 
vpn12725.inet.0: 6 destinations, 7 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.144.58.0/24     *[BGP/170] 00:46:02, MED 0, localpref 100, from 2.2.2.2
                      AS path: 65119 ?
                    > to 10.20.30.2 via em1.0, Push 18, Push 299792(top)
                    [BGP/170] 00:45:59, MED 0, localpref 100
                      AS path: 65119 65119 65119 65119 ?
                    > to 10.236.7.179 via em3.113
10.236.7.168/29    *[BGP/170] 00:48:21, localpref 100, from 2.2.2.2
                      AS path: I
                    > to 10.20.30.2 via em1.0, Push 18, Push 299792(top)
10.236.7.176/29    *[Direct/4] 00:48:56
                    > via em3.113
19.19.19.0/29      *[Direct/0] 00:48:58
                    > via em0.117
19.19.19.1/32      *[Local/0] 00:48:58
                      Local via em0.117
20.20.20.1/32      *[BGP/170] 00:46:54, MED 0, localpref 100
                      AS path: 65120 65120 65120 65120 I
                    > to 19.19.19.2 via em0.117
root@lahoreir3>

Above is happening because we have applied following route-map to firewall lahorcf3 to prepend the route 10.144.58.0/24 while not prepend when advertising via karacf3:

Configure route-map to not prepend 10.144.58.0/24 network as following:

!
access-list dmz standard permit 10.144.58.0 255.255.255.0 
!
route-map no-prepend permit 10
 match ip address dmz
!
!
router bgp 65119
  address-family ipv4 unicast
   neighbor 10.236.7.169 route-map no-prepend out
!

Similarly; confiture route-map to prepend 10.144.58.0/24 network on lahorcf3 as following:

!
access-list dmz standard permit 10.144.58.0 255.255.255.0 
!
route-map no-prepend permit 10
 match ip address dmz
 set as-path prepend 65119 65119 65119
!
!
router bgp 65119
  address-family ipv4 unicast
   neighbor 10.236.7.179 route-map prepend out
!

So if customer LAN PC1 or PC2 will try to ping the test machine 10.144.58.2/24, traffic will be take following path. Also note that I will keep the DR environment completely off, I will keep the lahorcf3 firewall interface Gi0/0.105 in admin down state as caution so it does not conflict.

PC1 > ESW1> R1> SW1> karachigw1> karais4 > karair3> karacf3> Service-vlan102 10.144.58.2/24

PC2 > ESW2> R2> R1> SW1>  karachigw1> karais4 > karair3> karacf3> Service-vlan102 10.144.58.2/24

Let’s see if PC1 can ping 10.144.58.2/24 behind karacf3:

PC1> ping 10.144.58.2
10.144.58.2 icmp_seq=1 timeout
84 bytes from 10.144.58.2 icmp_seq=2 ttl=62 time=65.638 ms
84 bytes from 10.144.58.2 icmp_seq=3 ttl=62 time=40.451 ms
84 bytes from 10.144.58.2 icmp_seq=4 ttl=62 time=42.351 ms
84 bytes from 10.144.58.2 icmp_seq=5 ttl=62 time=42.519 ms
PC1>

Let’s see customer LAN in Lahore office can also ping the 10.144.58.2/24 network:

PC2> ping 10.144.58.2
10.144.58.2 icmp_seq=1 timeout
10.144.58.2 icmp_seq=2 timeout
84 bytes from 10.144.58.2 icmp_seq=3 ttl=61 time=75.396 ms
84 bytes from 10.144.58.2 icmp_seq=4 ttl=61 time=54.293 ms
84 bytes from 10.144.58.2 icmp_seq=5 ttl=61 time=59.567 ms
PC2>

We can see IPSEC tunnel created on karachigw1:

karachigw1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.236.7.171    20.20.20.1      QM_IDLE           1001    0 ACTIVE
IPv6 Crypto ISAKMP SA

karachigw1#sh crypto ipsec sa
interface: FastEthernet0/0.116
    Crypto map tag: ptclmap, local addr 20.20.20.1
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (10.144.58.0/255.255.255.0/0/0)
   current_peer 10.236.7.171 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 3, #pkts encrypt: 3, #pkts digest: 3
    #pkts decaps: 3, #pkts decrypt: 3, #pkts verify: 3
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.171
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.116
     current outbound spi: 0xBF602470(3210749040)

     inbound esp sas:
      spi: 0xDBCC2FC5(3687591877)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 3, flow_id: SW:3, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4522540/3492)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
      spi: 0xBF602470(3210749040)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 4, flow_id: SW:4, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4522540/3492)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     outbound ah sas:
     outbound pcp sas:
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.179
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.116
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
     outbound ah sas:
     outbound pcp sas:
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (141.97.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (10.144.58.0/255.255.255.0/0/0)
   current_peer 10.236.7.171 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.171
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.116
     current outbound spi: 0x35480F0B(893914891)
     inbound esp sas:
      spi: 0x2E52E280(777183872)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 1, flow_id: SW:1, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4582889/3445)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
      spi: 0x35480F0B(893914891)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 2, flow_id: SW:2, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4582889/3445)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     outbound ah sas:
     outbound pcp sas:
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.179
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.116
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

No tunnel is created on lahoregw1 since this is Standby HSRP router at a moment:

lahoregw1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
IPv6 Crypto ISAKMP SA


lahoregw1#
karacf3# sh crypto isakmp sa
IKEv1 SAs:
   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1
1   IKE Peer: 20.20.20.1
    Type    : L2L             Role    : responder 
    Rekey   : no              State   : MM_ACTIVE 
There are no IKEv2 SAs

karacf3# 

So this proves our design working as expected. Now I am going to test if customer’s gateway router karachigw1 completely fails then traffic will pass via lahoregw1. To simulate this, I am going to shutdown internal and external physical interfaces of the karachigw1 router:

karachigw1#
karachigw1#sh ip int br
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            unassigned      YES NVRAM  up                    up      
FastEthernet0/0.116        18.18.18.2      YES NVRAM  up                    up      
FastEthernet1/0            172.168.1.2     YES NVRAM  up                    up     
FastEthernet1/1            unassigned      YES NVRAM  administratively down down    
FastEthernet2/0            unassigned      YES NVRAM  administratively down down    
FastEthernet2/1            unassigned      YES NVRAM  administratively down down    
SSLVPN-VIF0                unassigned      NO  unset  up                    up      
Loopback0                  20.20.20.1      YES NVRAM  up                    up      

karachigw1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
karachigw1(config)#int fa0/0
karachigw1(config-if)#sh
karachigw1(config-if)#int fa   
*Apr 11 18:56:54.655: %BGP-5-ADJCHANGE: neighbor 18.18.18.1 Down Interface flap
karachigw1(config-if)#int fa1/0
karachigw1(config-if)#
*Apr 11 18:56:56.619: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
karachigw1(config-if)#sh
karachigw1(config-if)#
*Apr 11 18:56:56.619: %ENTITY_ALARM-6-INFO: ASSERT INFO Fa0/0 Physical Port Administrative State Down 
karachigw1(config-if)#
*Apr 11 18:56:57.491: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 0 state Active -> Init
*Apr 11 18:56:57.619: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
karachigw1(config-if)#
*Apr 11 18:56:59.507: %LINK-5-CHANGED: Interface FastEthernet1/0, changed state to administratively down
*Apr 11 18:56:59.507: %ENTITY_ALARM-6-INFO: ASSERT INFO Fa1/0 Physical Port Administrative State Down 
karachigw1(config-if)#
*Apr 11 18:57:00.035: %TRACKING-5-STATE: 4 ip sla 3 reachability Up->Down
*Apr 11 18:57:00.507: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0, changed state to down
karachigw1(config-if)#
*Apr 11 18:57:09.207: %TRACKING-5-STATE: 5 ip route 10.236.7.168/29 reachability Up->Down
*Apr 11 18:57:10.035: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
*Apr 11 18:57:10.035: %TRACKING-5-STATE: 2 ip sla 2 reachability Up->Down
*Apr 11 18:57:10.127: %TRACKING-5-STATE: 3 list boolean and Up->Down
karachigw1(config-if)#

We can see the IP SLA kicked in as soon along with HSRP status changed. We have lahoregw1 as our Active HSRP router now:

lahoregw1#
*Apr 11 18:55:38.387: %HSRP-5-STATECHANGE: FastEthernet1/0 Grp 0 state Standby -> Active
lahoregw1#
lahoregw1#
lahoregw1#sh stan
lahoregw1#sh standby br
lahoregw1#sh standby brief 
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa1/0       0    100 P Active  local           unknown         172.168.1.1
lahoregw1#

Now let’s try to repeat our ping test from PC1 & PC2 to 10.144.58.2/24:

PC1> ping 10.144.58.2
10.144.58.2 icmp_seq=1 timeout
84 bytes from 10.144.58.2 icmp_seq=2 ttl=62 time=39.244 ms
84 bytes from 10.144.58.2 icmp_seq=3 ttl=62 time=70.384 ms
84 bytes from 10.144.58.2 icmp_seq=4 ttl=62 time=67.817 ms
84 bytes from 10.144.58.2 icmp_seq=5 ttl=62 time=56.503 ms
PC1>

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
ping 10.144.58.2
10.144.58.2 icmp_seq=1 timeout
84 bytes from 10.144.58.2 icmp_seq=2 ttl=61 time=52.481 ms
84 bytes from 10.144.58.2 icmp_seq=3 ttl=61 time=54.096 ms
84 bytes from 10.144.58.2 icmp_seq=4 ttl=61 time=39.883 ms
84 bytes from 10.144.58.2 icmp_seq=5 ttl=61 time=54.254 ms
PC2>

We have IPSEC tunnel established via lahoregw1 router to karacf3:

lahoregw1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.236.7.171    20.20.20.1      QM_IDLE           1001    0 ACTIVE
IPv6 Crypto ISAKMP SA
lahoregw1#

lahoregw1#sh crypto ipsec sa
interface: FastEthernet0/0.117
    Crypto map tag: ptclmap, local addr 20.20.20.1
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (10.144.58.0/255.255.255.0/0/0)
   current_peer 10.236.7.171 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.171
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0xECAD4797(3970779031)
     inbound esp sas:
      spi: 0x58990255(1486422613)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 3, flow_id: SW:3, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4456251/3550)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
      spi: 0xECAD4797(3970779031)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 4, flow_id: SW:4, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4456251/3550)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     outbound ah sas:
     outbound pcp sas:
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.179
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
     outbound ah sas:
     outbound pcp sas:
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (141.97.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (10.144.58.0/255.255.255.0/0/0)
   current_peer 10.236.7.171 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.171
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0x4D6F843E(1299153982)
     inbound esp sas:
      spi: 0xBCC92296(3167298198)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 1, flow_id: SW:1, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4531967/3532)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
      spi: 0x4D6F843E(1299153982)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 2, flow_id: SW:2, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4531967/3532)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     outbound ah sas:
     outbound pcp sas:
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.179
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas: 
     inbound pcp sas:
     outbound esp sas:
     outbound ah sas:
     outbound pcp sas:
lahoregw1#  



karacf3# sh crypto isakmp sa
IKEv1 SAs:
   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1
1   IKE Peer: 20.20.20.1
    Type    : L2L             Role    : responder 
    Rekey   : no              State   : MM_ACTIVE 
There are no IKEv2 SAs

karacf3#

Now bring back the interfaces on karachigw1 router and lets discuss the procedure to invoke the DR cloud firewall.

So once customer environment in primary cloud is lost due to any reason and customer cannot connect to the network 10.144.58.0/24, ISP or Cloud provide will be responsible to recover the protected machine in the DR cloud. Since protected machines in primary cloud will be recovered using same IP addressing in DR cloud, there will be no changes required at customer end. It will only need ISP or Cloud service provide to recover them as per the agreed SLA between the customer and ISP.

So the procedure would go like this.

  1. I will shutdown the karacf3 firewall so it will loose BGP peering within vpn12745 with PE karair3. So it will be effectively isolated. This is to simulate the primary cloud failure.
  2. This will cause customer’s primary gateway karachigw1 to change status to HSRP Standby since we have Object tracking and IP SLA configured to monitor the 10.236.7.171.
  3. I will also shutdown the loopback 0 interface on karachigw1 because we are still learning 20.20.20.1/32 from karachigw1 since BGP peering between karahigw1 and PE router karair3 is still intact. Shutting loopback 0 will allow IPSEC tunnel to be establish between lahoregw1 and lahorc3 in this situation.
  4. Traffic from customer’s LAN in Lahore & Karachi will take following path:

PC1 > ESW1 > R1 > SW1 > SW2 > lahoregw1 > Lahoris4 >  lahorir3 > lahorcf3> Service-vlan103 10.144.58.2/24

PC2 > ESW2 > R2 > SW2 > lahoregw1 > lahoris4 >  lahorir3 > lahorcf3> Service-vlan103 10.144.58.2/24

Let’s ping from both PC1 and PC2 and check the IPSEC tunnel status in cloud and customer environment:

PC1> ping 10.144.58.2
10.144.58.2 icmp_seq=1 timeout
84 bytes from 10.144.58.2 icmp_seq=2 ttl=62 time=58.001 ms
84 bytes from 10.144.58.2 icmp_seq=3 ttl=62 time=59.092 ms
^C

PC1> sh
NAME   IP/MASK              GATEWAY           MAC                LPORT  RHOST:PORT
PC1    141.97.1.10/16       141.97.1.1        00:50:79:66:68:04  10092  127.0.0.1:10093
       fe80::250:79ff:fe66:6804/64
PC1> 

lahoregw1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.236.7.179    20.20.20.1      QM_IDLE           1005    0 ACTIVE
IPv6 Crypto ISAKMP SA


lahoregw1#
lahoregw1#sh crypto ipsec sa
interface: FastEthernet0/0.117
    Crypto map tag: ptclmap, local addr 20.20.20.1
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (10.144.58.0/255.255.255.0/0/0)
   current_peer 10.236.7.179 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 5, #recv errors 0
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.171
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
     outbound ah sas:
     outbound pcp sas:
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.179
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
     outbound ah sas:
     outbound pcp sas:
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (141.97.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (10.144.58.0/255.255.255.0/0/0)
   current_peer 10.236.7.179 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 37, #pkts encrypt: 37, #pkts digest: 37
    #pkts decaps: 37, #pkts decrypt: 37, #pkts verify: 37
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 7, #recv errors 0
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.171
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0x0(0)
     inbound esp sas:
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
     outbound ah sas:
     outbound pcp sas:
     local crypto endpt.: 20.20.20.1, remote crypto endpt.: 10.236.7.179
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0.117
     current outbound spi: 0xA620859(174196825)
     inbound esp sas:
      spi: 0x5AFAF123(1526395171)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 13, flow_id: SW:13, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4598244/3535)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     inbound ah sas:
     inbound pcp sas:
     outbound esp sas:
      spi: 0xA620859(174196825)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Tunnel, }
        conn id: 14, flow_id: SW:14, crypto map: ptclmap
        sa timing: remaining key lifetime (k/sec): (4598244/3535)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE
     outbound ah sas:
     outbound pcp sas:
lahoregw1#

So we have successfully tested the DR Cloud as well and every things seems to be working as per design.

Let’s also verify the R1 & R2 failover scenario. We have R1 as an Active HSRP router at a moment while R2 is Standby:

R1#
*Mar  1 00:00:20.819: %HSRP-5-STATECHANGE: FastEthernet2/0 Grp 1 state Standby -> Active
R1#
R2#
*Mar  1 00:00:26.755: %HSRP-5-STATECHANGE: FastEthernet2/0 Grp 1 state Speak -> Standby
R2#

I am going to shutdown the R1 interface Fa2/0 so that R1 will change its status to Standby. After doing that we should still be able to ping the HSRP VIP address of gateway routers karachigw1 and lahoregw1:

R1#sh standby brief 
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa2/0       1    120 P Active  local           172.168.1.6     172.168.1.4
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int fa2/0
R1(config-if)#sh
R1(config-if)#
*Mar  1 00:06:38.659: %HSRP-5-STATECHANGE: FastEthernet2/0 Grp 1 state Active -> Init
*Mar  1 00:06:38.759: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 172.168.1.6 (FastEthernet2/0) is down: interface down
R1(config-if)#
*Mar  1 00:06:40.671: %LINK-5-CHANGED: Interface FastEthernet2/0, changed state to administratively down
*Mar  1 00:06:41.671: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet2/0, changed state to down
R1(config-if)#
R2#
*Mar  1 00:06:28.571: %HSRP-5-STATECHANGE: FastEthernet2/0 Grp 1 state Standby -> Active
R2#
*Mar  1 00:06:42.995: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 172.168.1.5 (FastEthernet2/0) is down: holding time expired
R2#

 

Let’s PING the HSRP VIP address 172.168.1.1:

PC1 : 141.97.1.10 255.255.0.0 gateway 141.97.1.1
PC1> ping 172.168.1.1
172.168.1.1 icmp_seq=1 timeout
84 bytes from 172.168.1.1 icmp_seq=2 ttl=252 time=44.984 ms
84 bytes from 172.168.1.1 icmp_seq=3 ttl=252 time=50.125 ms
84 bytes from 172.168.1.1 icmp_seq=4 ttl=252 time=77.711 ms
84 bytes from 172.168.1.1 icmp_seq=5 ttl=252 time=71.789 ms
PC1>

 

This is also working as expected.

Stingray/Brocade Software Uprade 10.4r1

Upgrading a Stingray Traffic Manager is easy either via CLI or GUI. Following are easy steps to upgrade via the Administration Interface on STM:

Upgrade Procedure via GUI:
1. Obtain the new installation package ( for example ZeusTM_104r1_Linux-x86_64.tgz)

2. Navigate to the System > Traffic Managers page on the Admin Interface of the Traffic Manager you intend to upgrade.

3. Click the Upgrade button and upload the new installation package

4. Follow the instructions to apply the software upgrade.

*Software version mismatch alarm is expected if your upgrading a cluster. This alarm will clear itself once complete cluster is running on same version.

Roll back Procedure via GUI:
1. Navigate to the System> Traffic Managers page on the the Admin Interface of the Traffic Manager you intend to upgrade.

2. Click the rollback button and select the old installation package for example release 9.9.

  1. Follow the instructions to apply the software upgrade.

 

Upgrade procedure via CLI:

  1. Upload and extract the software package for xample ZeusTM_104r1_Linux-x86_64.tgz in /root
  2. Get into the cd ZeusTM_104r1_Linux-x86_64
  3.  Run the installation script ./zinstall
  4. Select upgrade and allow to complete.

Through the web interface verify that the load balancer has returned to production operation. Warnings are expected on the Cluster Diagnosis due to the version mismatch within the cluster.

 

Rollback procedure via CLI

On each load balancer in turn, as required:

  1. Get into the cd /usr/local/zeus
  2. Run the rollback script ./zxtm/bin/rollback
  3. Select version the older version for example 9.9
  4. Allow the downgrade process to run to completion

 

AWS: Storage

AWS Storage services provide different options of storage of options based on usage:

1. Amazon S3
a. This is also called simple storage that is durable and scaleable.
b. Provides 99.999999999% durability
c. Internet scale storage via API
d. Used for Static Video/Images.

2. Amazon Elastic Block Storage (EBS)
a. Block storage for use with Amazon EC2
b. Automatically replicated to the AZ provides high availability.
c. Used for low latency application and where data consistency is required.

3. Amazon Glacier
a. Storage for archiving and backup.
b. Low cost roughly 1 cent per GB per Month

4. AWS Storage Gateway
a. This storage service allows customer to connect their on-premises application to connect to the AWS storage either S3 or Glacier.
b. Integrates on-premises IT and AWS storage.
c. Securley upload data to AWS cloud for cost effective backup and rapid disaster recovery.

5. Amazon Elastic File System (EFS)
a. Provides interface to configure file system for EC2 instances.

6. AWS Import/Export Snowball:
a. Device designed by AWS to transfer large amount of Data that cannot be transfere via internet due to size, amout and security.
b. Data stores in encrypted format in the device
c. Device is trackable
d. Once recived by the Cloud service provide, data is then transfered in customer’s AWS storage.

7. AWS Cloudfront:
a. This is content delivery network that cache stored data at edge locations
b. Customer access the data from closest edge location, hence achieve low latency
c. Cloudfront use S3 stroage or may use third party storage as well.

AWS Storage key features:

a. Low cost: Pay only for the storage and provision performance.
b. Elastic: Scale storage capacity as computing requirement change
c. Flexible: Choice of storage types
d. Secure: Encrypt EBS storage. Set permission on S3 storage.

Compare Amaon S3 vs EBS:

S3 Object Storage:
a. Storage for Internet.
b. Store and retrive any amount of data and used for Static images and video files.
c. Latency is high when retrieving the stored data.
d. Not allowed to modify the stored data
e. S3 stroage does not get replicated.

EBS Storage:
a. Behaves like hard drive, you can create partetion, format, and boot your operation system off.
b. Stored data can be modified and gets replicated to AZ automatically.
c. Used for low latency data for example database/applitions/website etc.
d. Two types of Volume used with EBS. Standard type where I/O perforamnce is not critical. Provisioned IOPS where performance is critical example databases.

AWS: Compute

Compute Services address the computational needs of the users in the cloud. There are following features available under AWS Compute Services:

1. Amazon Elastic Compute Cloud or EC2:
a. Virutal computing environment ( VM instance)
b. Preconfigured template for instances (Amazon Machine Image AMI that package the operating system and various softwares),
c. Selection of different Operating System, Memory size, storage & networking capacity
d. Amazon EC2 can use feature of Auto-Scaling and Elastic Load Balancing between multiple instance of VM.

2. AWS Lamda:
a. Runs backend code in case of any event for example website click and manages the compute resources.

3. Amazon EC2 Container Service (ECS):
a. It is container mangement service that supports docker container.
b. This feature allow you run application on managed cluster of EC2 instances.

Amazon Compute Services key benefits include:

a. Elasticity: Scale up and down the EC2 instances based on computing requirement.
b. Flexibility: Different CPU types, Operating Systems, Memory and Networking configuration for EC2.
c. Secure: Complete control of compute resouces for the customers to mangage themself.
d. Cost: Pay what you use.

Following types of EC2 instances available:

a. General Purpose M4
b. Compute Optimized C4 (high processing requirement)
c. Memory Optimized R3 (memory intensive applications)
d. Burstable performance T2
e. Graphics & GPU compute application GPU G2
f. Storage Optimized L2 (very fast SSD-backed instance storage)
g. Dense Storage D2 (High throughput & low price per disk)

Cost model for EC2 instances:

a. Dedicated: EC2 instances are lauched on dedicated customer hardware
b. On-Demand: Pay per hour. No long term contract.
c. Reserved: Pay-upfront for steady workflow. cheaper then On-Demand.
d. Spot: Bid for unused EC2 capacity, if you win you will get the EC2 instance.

4. Elastic Load Balancing (ELB):
Automatically distribute incoming traffic to multiple EC2 instances to achieve fault tolerance.

a. Works with Auto-Scaling feature of EC2 to add/remove instances based on scaling activities.
b. Health monitor of EC2 instances
c. Works with Virtual Private Cloud (VPC) to network security features.

Active/Standby Data Centre Network Design using GNS3/Virtualbox/JUNOS and Cisco – Part-4

You can see the previous tutorial Part-3 of this design series to understand the work I have done so far.

I this tutorial, I am going to add vpn12745 at both lahorir3 and karair3 PE routers. This is stretched VPN so this will need to be configured at both PE lahorir3 & karair3. vpn12745 will host incoming routes from subnets behind the virtual data center firewalls. I will use eBGP between karacf3/lahorcf3 Cisco ASAv firewalls and karair3/lahorir3 PE routers in vpn12745 to allow customer’s offices to reach their subnets in their virtual cloud hosted in Karachi and Lahore data centers.

I will add 2nd VPN vpn12725 at both PE routers (karair3/lahorir3). This is again stretched VPN and will carry incoming routes from two different locations of customer’s office (PTCL Karachi and Lahore office). These customer’s offices are connected via a WAN link provider to the PE routers (karair3/lahorir3). Customer use their routers (lahoregw1 and karachigw1) to establish eBGP connectivity to these PE routers (karair3/lahorir3) to reach their subnets in virtual clouds dc-lahore/dc-karachi, hosted in Karachi & Lahore data centres.

As part of adding above two VPN, I will also leak routes between above two vpn (vpn12725 and vpn12745) and filter the routes passing via these two VPN.

Final Topology

 

LAN users in customer’s offices in Lahore & Karachi will be connected to the Layer-3 Ethernet switch (ESW1 & ESW2). I am using 3700 series switch in this design. R1 & R2 will provide failover protection hence HSRP will be used between R1 & R2. Similarly; gateway routers karachigw1 & lahoregw1 will also provide failover protection and run HSRP between them.

Once configured, customer’s router lahoregw1/karachigw1 should be able to ping their relevant firewall interfaces in their data centers karacf3 (dc-karachi interface service vlan 102) and lahorcf3 (dc-lahore interface service vlan 103).

So I will go through following steps to build further as discussed above.

  1. Add vpn12745 layer-3 vpn on karair3/lahorir3 and establish eBGP peering with karair3/lahorir3 virtual data centre firewalls.
root@karair3> show configuration groups vpn12745 | display set 
set groups vpn12745 interfaces em3 unit 112 description vpn=12745
set groups vpn12745 interfaces em3 unit 112 vlan-id 112
set groups vpn12745 interfaces em3 unit 112 family inet address 10.236.7.169/29
set groups vpn12745 interfaces em3 unit 112 family mpls
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 0 then next term
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from protocol static
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from protocol bgp
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from interface em3.112
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 then community add vpn12745
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 0 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 0 then preference 4
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from protocol static
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from protocol bgp
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from community vpn12725
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-import-ce term 1 from route-filter 0.0.0.0/0 exact
set groups vpn12745 policy-options policy-statement vpn12745-import-ce term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-import-ce term 2 then reject
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 from protocol static
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 from protocol bgp
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 2 then reject
set groups vpn12745 policy-options community vpn12745 members target:65000:12745
set groups vpn12745 policy-options community vpn12725 members target:65000:12725
set groups vpn12745 routing-instances vpn12745 description vpn=12745
set groups vpn12745 routing-instances vpn12745 instance-type vrf
set groups vpn12745 routing-instances vpn12745 interface em3.112
set groups vpn12745 routing-instances vpn12745 route-distinguisher 65000:12745
set groups vpn12745 routing-instances vpn12745 vrf-import vpn12745-import-vrf
set groups vpn12745 routing-instances vpn12745 vrf-export vpn12745-export-vrf
set groups vpn12745 routing-instances vpn12745 vrf-table-label
set groups vpn12745 routing-instances vpn12745 routing-options auto-export
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 type external
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 import vpn12745-import-ce
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 authentication-key cisco123
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 export vpn12745-export-ce
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 peer-as 65119
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 neighbor 10.236.7.171
root@lahorir3> show configuration groups vpn12745 | display set 
set groups vpn12745 interfaces em3 unit 113 description vpn=12745
set groups vpn12745 interfaces em3 unit 113 vlan-id 113
set groups vpn12745 interfaces em3 unit 113 family inet address 10.236.7.177/29
set groups vpn12745 interfaces em3 unit 113 family mpls
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 0 then next term
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from protocol static
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from protocol bgp
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 from interface em3.113
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 then community add vpn12745
set groups vpn12745 policy-options policy-statement vpn12745-export-vrf term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 0 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 0 then preference 4
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from protocol static
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from protocol bgp
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 from community vpn12725
set groups vpn12745 policy-options policy-statement vpn12745-import-vrf term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-import-ce term 1 from route-filter 0.0.0.0/0 exact
set groups vpn12745 policy-options policy-statement vpn12745-import-ce term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-import-ce term 2 then reject
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 from protocol direct
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 from protocol static
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 from protocol bgp
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 1 then accept
set groups vpn12745 policy-options policy-statement vpn12745-export-ce term 2 then reject
set groups vpn12745 policy-options community vpn12745 members target:65000:12745
set groups vpn12745 policy-options community vpn12725 members target:65000:12725
set groups vpn12745 routing-instances vpn12745 description vpn=12745
set groups vpn12745 routing-instances vpn12745 instance-type vrf
set groups vpn12745 routing-instances vpn12745 interface em3.113
set groups vpn12745 routing-instances vpn12745 route-distinguisher 65000:12745
set groups vpn12745 routing-instances vpn12745 vrf-import vpn12745-import-vrf
set groups vpn12745 routing-instances vpn12745 vrf-export vpn12745-export-vrf
set groups vpn12745 routing-instances vpn12745 vrf-table-label
set groups vpn12745 routing-instances vpn12745 routing-options auto-export
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 type external
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 import vpn12745-import-ce
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 authentication-key cisco123
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 export vpn12745-export-ce
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 peer-as 65119
set groups vpn12745 routing-instances vpn12745 protocols bgp group vpn12745 neighbor 10.236.7.179

2. Add vpn12725 layer-3 vpn on karair3/lahorir3 and establish eBGP peering with customer’s routers karachigw1 and lahoregw1.

root@lahorir3> show configuration groups vpn12725 | display set    
set groups vpn12725 interfaces em0 vlan-tagging
set groups vpn12725 interfaces em0 unit 117 description vpn=12725
set groups vpn12725 interfaces em0 unit 117 vlan-id 117
set groups vpn12725 interfaces em0 unit 117 family inet address 19.19.19.1/29
set groups vpn12725 interfaces em0 unit 117 family mpls
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 0 then next term
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from protocol static
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from protocol bgp
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from interface em0.117
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 then community add vpn12725
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 0 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 0 then preference 4
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from protocol static
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from protocol bgp
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from community vpn12725
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from community vpn12745
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-import-ce term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-import-ce term 2 then reject
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 from protocol static
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 from protocol bgp
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 2 then reject
set groups vpn12725 policy-options community vpn12725 members target:65000:12725
set groups vpn12725 policy-options community vpn12745 members target:65000:12745
set groups vpn12725 routing-instances vpn12725 description vpn=12725
set groups vpn12725 routing-instances vpn12725 instance-type vrf
set groups vpn12725 routing-instances vpn12725 interface em0.117
set groups vpn12725 routing-instances vpn12725 route-distinguisher 65000:12725
set groups vpn12725 routing-instances vpn12725 vrf-import vpn12725-import-vrf
set groups vpn12725 routing-instances vpn12725 vrf-export vpn12725-export-vrf
set groups vpn12725 routing-instances vpn12725 vrf-table-label
set groups vpn12725 routing-instances vpn12725 routing-options auto-export
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 type external
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 import vpn12725-import-ce
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 authentication-key cisco123
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 export vpn12725-export-ce
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 peer-as 65120
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 neighbor 19.19.19.2
root@karair3> show configuration groups vpn12725 | display set    
set groups vpn12725 interfaces em1 vlan-tagging
set groups vpn12725 interfaces em1 unit 116 description vpn=12725
set groups vpn12725 interfaces em1 unit 116 vlan-id 116
set groups vpn12725 interfaces em1 unit 116 family inet address 18.18.18.1/29
set groups vpn12725 interfaces em1 unit 116 family mpls
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 0 then next term
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from protocol static
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from protocol bgp
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 from interface em1.116
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 then community add vpn12725
set groups vpn12725 policy-options policy-statement vpn12725-export-vrf term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 0 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 0 then preference 4
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from protocol static
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from protocol bgp
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from community vpn12725
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 from community vpn12745
set groups vpn12725 policy-options policy-statement vpn12725-import-vrf term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-import-ce term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-import-ce term 2 then reject
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 from protocol direct
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 from protocol static
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 from protocol bgp
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 1 then accept
set groups vpn12725 policy-options policy-statement vpn12725-export-ce term 2 then reject
set groups vpn12725 policy-options community vpn12725 members target:65000:12725
set groups vpn12725 policy-options community vpn12745 members target:65000:12745
set groups vpn12725 routing-instances vpn12725 description vpn=12725
set groups vpn12725 routing-instances vpn12725 instance-type vrf
set groups vpn12725 routing-instances vpn12725 interface em1.116
set groups vpn12725 routing-instances vpn12725 route-distinguisher 65000:12725
set groups vpn12725 routing-instances vpn12725 vrf-import vpn12725-import-vrf
set groups vpn12725 routing-instances vpn12725 vrf-export vpn12725-export-vrf
set groups vpn12725 routing-instances vpn12725 vrf-table-label
set groups vpn12725 routing-instances vpn12725 routing-options auto-export
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 type external
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 import vpn12725-import-ce
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 authentication-key cisco123
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 export vpn12725-export-ce
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 peer-as 65120
set groups vpn12725 routing-instances vpn12725 protocols bgp group vpn12725 neighbor 18.18.18.2 passive
  1. Customer aggregation switches karais3 and lahoris3 are configured as following:
!
interface FastEthernet1/0
 description to_karair3:em1
 switchport mode trunk
 duplex full
 speed 100
!
interface FastEthernet1/1
 description to_karachigw1:f0/0
 switchport mode trunk
 duplex full
 speed 100
!
         
karais3#sh int trunk 
Port      Mode         Encapsulation  Status        Native vlan
Fa1/0     on           802.1q         trunking      1
Fa1/1     on           802.1q         trunking      1
Port      Vlans allowed on trunk
Fa1/0     1-4094
Fa1/1     1-4094
Port      Vlans allowed and active in management domain
Fa1/0     1,116
Fa1/1     1,116
Port      Vlans in spanning tree forwarding state and not pruned
Fa1/0     1,116
Fa1/1     1,116
!
interface FastEthernet1/0
 description to_lahorir3:em0
 switchport mode trunk
 no ip address
 duplex full
 speed 100
!         
interface FastEthernet1/1
 description to_lahoregw1:fa0/0
 switchport mode trunk
 no ip address
 duplex full
 speed 100
!

lahoris3#sh int trunk 
Port      Mode         Encapsulation  Status        Native vlan
Fa1/0     on           802.1q         trunking      1
Fa1/1     on           802.1q         trunking      1
Port      Vlans allowed on trunk
Fa1/0     1-1005
Fa1/1     1-1005
Port      Vlans allowed and active in management domain
Fa1/0     1,117
Fa1/1     1,117
Port      Vlans in spanning tree forwarding state and not pruned
Fa1/0     1,117
Fa1/1     1,117
  1. Configuration of customer managed routers karachigw1/lahoregw1 in their office in Karachi and Lahore configured as following:
karachigw1#

!
interface FastEthernet0/0.116
 description to_karair3
 encapsulation dot1Q 116
 ip address 18.18.18.2 255.255.255.248
!
!
router bgp 65120
 no synchronization
 bgp log-neighbor-changes
 network 18.18.18.0 mask 255.255.255.248
 neighbor 18.18.18.1 remote-as 65000
 neighbor 18.18.18.1 password cisco123
 neighbor 18.18.18.1 soft-reconfiguration inbound
 no auto-summary
!         
ip route 10.141.33.96 255.255.255.240 18.18.18.1
ip route 18.18.18.0 255.255.255.248 Null0
!
lahoregw1#
!
interface FastEthernet0/0.117
 description to_lahoris3
 encapsulation dot1Q 117
 ip address 19.19.19.2 255.255.255.248
!

router bgp 65120
 no synchronization
 bgp router-id 19.19.19.2
 bgp log-neighbor-changes
 network 19.19.19.0 mask 255.255.255.248
 neighbor 19.19.19.1 remote-as 65000
 neighbor 19.19.19.1 password cisco123
 neighbor 19.19.19.1 soft-reconfiguration inbound
 no auto-summary
!
ip forward-protocol nd
ip route 10.141.212.144 255.255.255.240 19.19.19.1
ip route 19.19.19.0 255.255.255.248 Null0
!

Now let’s try to ping from karachigw1 and lahoregw1 customer managed routers to the Cloud Firewall interfaces karacf3/lahorcf3 in dc-karachi & dc-lahore respectively.

lahoregw1#ping 10.236.7.179
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.236.7.179, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/10/12 ms
lahoregw1#
karachigw1#ping 10.236.7.171
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.236.7.171, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/11/16 ms
karachigw1#

We have now IP reachability from customer’s offices in Lahore & Karachi to their dc-lahore & dc-karachi firewalls.

Netscreen useful commands

Here is list of some useful Netscreen commands:

1. Search for the virtual firewall context from main context:
get conf | i context-name

2. Get into the vsys context:
enter vsys context-name

3. Identificy policy action is denying/allowing packing from source 
to destion using service:
get policy src-ip x.x.x.x dst-ip x.x.x.x "service name eg HTTPS"

4. Define an IP addresss in a zone:
set address "zone-name" "address-book-name" x.x.x.x 255.255.255.255

5. Define a group in a zone and add an address into group:
set group address "zone-name" "Public_Networks"
set group address "zone-name" "Public_Networks" add "address-book-name"

6. Define a service/port and add it into a group (optional):
set service "tcp_41458" protocol tcp src-port 0-65535 dst-port 41458-41458
set group service "TCP_Citrix" add "tcp_41458"

7. Define a policy to permit https traffic from inside zone to outside 
zone and NAT the outgoing traffic and log as well. Let Policy id auto created:
set policy top from "inside" to "outside" "inside-address-book-name" "outside-address-book-name" "HTTPS" nat src permit log

8. Define a policy id manualy:
set policy id 102 from "inside" to "outside" "inside-address-book-name" "outside-address-book-name" "HTTPS" permit log

9. Add an additional service of "DNS" in existing policy 102:
set service "DNS" protocol tcp src-port 0-65535 dst-port 53-53
set policy id 102
set service "DNS"
exit

10. Unset an existing policy 102
unset policy id 102

11. Check general overview of VPN configured
netscreen(M)-> get vpn

12. Confirm Phase 1:
To confirm whether IKE has been successful you can run the following command. 
You may find though that there is no IKE cookie but there is a Phase 2 Security Assicoation. 
This is due to the Phase 1 IKE lifetime being set to a value less then the IKE Phase 2 lifetime. 
netscreen(M)->get ike cookie | i [remote peer ip]
81182f/0003, [REMOTE-PEER-IP]:500->[LOCAL-PEER-IP]:500, PRESHR/grp5/AES128/SHA, xchg(5) (VPN-gateway/grp-1/usr-1)

13. Confirm Phase 2:
From the get sa command you can see the status and various details of the Security Assiociations. 
The section below which is highlighted in bold shows the status of the vpn tunnel (left) and the 
status of the VPN monitor (right). In this case the VPN tunnel is active and the VPN monitor is 
dashed out as it isnt enabled.
netscreen(M)-> get sa | i [peer ip]
00000007<       [peer ip]  500 esp:3des/md5  zbcA14zz  3317 unlim A/-    22 0
00000007>       [peer ip]  500 esp:3des/md5  fbcb64ee  3317 unlim A/-    -1 0

Using the SA ID we can confirm additional details of the Phase 2 SA.

netscreen(M)-> get sa id 0x00000007
index 49, name Example, peer gateway ip [remote peer]. vsys<Root>
auto key. policy node, tunnel mode, policy id in:<10104> out:<10103> vpngrp:<-1>. sa_list_nxt:<-1>.
tunnel id 662, peer id 52, NSRP Active. Vsd 0   site-to-site. Local interface is ethernet5 
<[local peer]>.
  esp, group 0, a256 encryption, sha1 authentication
  autokey, IN active, OUT active
  monitor<0>, latency: 0, availability: 0
  DF bit: clear
  app_sa_flags: 0x2067
  proxy id: local 0.0.0.0/0.0.0.0, remote 0.0.0.0/0.0.0.0, proto 0, port 0
  ike activity timestamp: 590051543
nat-traversal map not available
incoming: SPI 9j32882e, flag 00004000, tunnel info 40000296, pipeline
  life 86400 sec, 19761 remain, 0 kb, 0 bytes remain
  anti-replay on, last 0xb6840, window 0xffffffff, idle timeout value <0>, idled 0 seconds
  next pak sequence number: 0x0
outgoing: SPI 7bz2a942, flag 00000000, tunnel info 40000296, pipeline
  life 86400 sec, 19761 remain, 0 kb, 0 bytes remain
  anti-replay on, last 0x0, window 0x0, idle timeout value <0>, idled 0 seconds
  next pak sequence number: 0x89j9c

14. Netscreen - Rekeying a VPN / Clearing the SA`s
In order to rekey a Netscreen VPN you will need to either clear the phase 1 or phase 2 "keys" from the gateway. 
Phase 1 being the IKE cookies and phase 2 being the SA`s (Security Association).

To see an overview of your VPN`s run the command:
get vpn

In order to find the current IKE Cookies or SA`s, run either of the following commands,
get ike cookies 
get sa active

To clear either of these run either or of the following commands:
clear ike-cookie [gateway ip] 
clear sa [id] 

Below shows you an example of clear a VPN`s SA`s,

ns5gt-> get sa active
Total active sa: 1
total configured sa: 1
HEX ID    Gateway         Port Algorithm     SPI      Life:sec kb Sta   PID vsys
00000007<       10.1.1.25  500 esp:3des/md5  ef1d167f  3317 unlim A/-    22 0
00000007>       10.1.1.25  500 esp:3des/md5  fbcb64ee  3317 unlim A/-    -1 0

ns5gt-> clear sa 00000007
ns5gt-> get sa active
Total active sa: 1
total configured sa: 1
HEX ID    Gateway         Port Algorithm     SPI      Life:sec kb Sta   PID vsys
00000007<       10.1.1.25  500 esp:3des/md5  ef1d1680  3592 unlim A/-    22 0
00000007>       10.1.1.25  500 esp:3des/md5  bd1cbef7  3592 unlim A/-    -1 0

The main thing to ensure is that you show only the active sa`s as the firewall will not let you clear 
inactive sa`s. You can tell that they are active as the "Sta" (State) is A/- which is active. 
Also note that the Hex ID was used when using the `clear sa` command.

15. RUNNING A DEBUG
Here we will run a debug so we can obtain a more verbose view of what is happening to our traffic.
netscreen(M)-> set ff src-ip [local endpoint] dst-ip [remote endpoint] 
netscreen(M)-> undebug all
netscreen(M)-> clear db
netscreen(M)-> debug ike basic
netscreen(M)-> debug flow basic
netscreen(M)-> get db str
!
!
Permitted by policy 109
  No src xlate   choose interface ethernet5 as outgoing phy if
  check nsrp pak fwd: in_tun=0xffffffff, VSD 0 for out ifp ethernet5
  vsd 0 is active
  no loop on ifp ethernet5.
  session application type 0, name None, nas_id 0, timeout 60sec
  service lookup identified service 0.
  flow_first_final_check: in , out 
  existing vector list 25-6870620.
  Session (id:127345) created for first pak 25
  flow_first_install_session======>
  cache mac in the session
  make_nsp_ready_no_resolve()
  search route to (ethernet5, [remote endpoint]->[local endpoint]) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet2
  [Dest] 10.route [local endpoint]->[next hop], to ethernet2
  route to [next hop]
  nsrp msg sent.
  flow got session.
  flow session id 127345
  vsd 0 is active
  skipping pre-frag 
  going into tunnel 40000266.
  flow_encrypt: pipeline.
chip info: DMA. Tunnel id 00000266
(vn2)  doing ESP encryption and size =64
ipsec encrypt prepare engine done
ipsec encrypt set engine done
ipsec encrypt engine released
ipsec encrypt done
    put packet(557a0f0) into flush queue.
    remove packet(557a0f0) out from flush queue.

If the tunnel does not come up you can use the following debug:
netscreen(M)-> ike detail set sa-filter [IP] 

16. EVENT LOGS
In addition to check the Logs that the traffic is being passed you can check for 
Phase 1 and Phase 2 errors from the devices event logs. 

netscreen(M)-> get event include [peer ip]