Resetting the VMWare vCenter appliance (VCSA) root password

Ok this is strange problem I came across during my VMWare lab setup. For some reason the password I’ve setup during my vcenter appliance setup is not detecting. Yes I waited enough time for appliance to bootup Smile 

So here I’m wondering how to move to next step without wiping and starting my test lab from the scratch. Based on my research around found out we can reset the root password on Linux appliance. This going to be a post which I will be helpful for me as well as for yo all when struggling through root password issues in VCSA

To provide some background I’m testing VMWare Esxi and vCenter (VCSA) 6.7 setup running on VMWare workstation 14. For this lab setup I did import the VCSA OVA file directedly into the VMWare workstation instead of setting up inside the ESXi server (yes you can do that Smile)

Please note VMware using their own OS called “Photon OS” for their VCSA appliance. 

Ok for the steps guide now.

1. Take a snapshot or backup or your VCSA appliance. Restart the VM and hit letter “e” in your keyboard when you see the “Photon OS splash screen. This will take you to the GRUB boot menu. In the end of first sentence append the parameter rw init=/bin/bash

image

2. After that press F10 to continue the boot. After few seconds you’ll be prompted with  root login. Type passwd. You’ll be prompted to enter the new password. Type the new password and repeat the again for verification. If you’re successful you’ll be prompted.

image 

image

3. Once the steps are completed you can type reboot –f and let the system restart. After that once the system reboot try connecting to vcsa login screen with the new password.

Hope this small steps will save you lot of time. From what I heard if there is new updates to the Photon OS there is a possibility password we setup might not work. Remember these steps to overcome that.

Kindly note I’ve only test this on VMware 6.7 version only. I do believe same will work on 6.5 version as well.

Advertisements

How to migrate Public IP between Azure VM’s

This article created based on a challenge I faced on migrating Public IP [Static] to a different VM. There are many scenarios why you might want to keep static public IP to a Azure VM (Iaas). Despite being said to leverage DNS names we know in practical world static IP still wins

Smile

In this scenario I had a challenge of my customer’s VM has been attacked by ransomware. Lucky we had taken full backup of the VM. First tried restoring the disks to the same VM but problem still exists. Next solution is restoring the backup to a new VM (entire VM restore) How to do that you can find here.

Below video will share you how I manage to resolve the problem.

https://www.youtube.com/watch?v=-R63rlspMJU

Handle Windows servers with Windows Admin Center aka “Project Honolulu”

Tis project has been initially announced on year 2017 on Ignite event. At that time it just gave excitement for server admins who saw this as Swiss army knife to managed the different version of Windows servers. Along with time Microsoft project team has been working very closely with MVP’s and general public getting the feedback how they should be improving this project.

In Year 2018 this has been announced on GA level. So what is Windows Admin Center and it’s advantages? To address that I’ll share the exact information shared by Windows Server Team,

  • Simple and modern management experience: Windows Admin Center is a lightweight, browser-based GUI platform and toolset for IT admins to remotely manage Windows Server and Windows 10 machines.
  • Hybrid capabilities: Windows Admin Center can manage Windows Server and Windows 10 instances anywhere including physical systems, virtual machines on any hypervisor, or running in any cloud. Connect to the cloud with optional value-added features like integration with Azure Site Recovery for protecting your virtual machines, and support for Azure Active Directory to control access with multi-factor authentication.
  • Integrated toolset: Rather than switching between several different tools and contexts, with Windows Admin Center you get a holistic overview of your resources and the ability to dig into granular details. In addition to server and client machines, it allows you to manage failover clusters and hyper-converged infrastructure (HCI) deployments.
  • Designed for extensibility: We’ve been working with early-adopter partners to refine the extension development experience in a private preview of our SDK. That means soon you’ll be able to extend Windows Admin Center’s capabilities to 3rd-party solutions. For example, you’ll start to see 3rd party hardware vendors use Windows Admin Center to provide management of their own hardware.

For me it’s really interesting when Jeff Woolsey mention in the Azure Cloud Summit Singapore you can manage your Azure resources as well as on-premise resources from single console.

I’ve already went ahead and tested this on my test lab cross checking server 2012 R2 to all the way to Server vNext (aka Server 2019) version. I’m pretty amazed how simply product can be setup and used. Kindly note there are couple of modes you can setup Windows Admin Center to manage your servers.

PS: Server 2012 R2 had problems due to WMF 5.1 not availbility. This has been documented on Microsoft docs and easily can be fixed.

image

image

In my case I went ahead with setting everything on a Windows 10 Enterprise VM. I think most of the production environments in Sri Lanka would be fine with Singe Gateway mode. If you’re a service provider managing lot of customer resources then it’s fine to move ahead with Windows Admin Center on failover cluster to provide additional resiliency.

In my test lab I’ve both Server 2016 and Server 2019 in command level. Whatever said and done it’s not that easy when you miss the GUI Smile. Windows Admin Center resolves that problem like a charm. I’ve managed to connect to both servers seamless level and manage them remotely. Only problem I came across is when I tried to enable remote access to both servers from Windows Admin Center. Finally I end up using sconfig and enable remote desktop access. Soon after that I’ve managed to get RDP to both servers for console access.

Few screenshots of my small lab environment,

image

Managing Server 2019 (Yes Microsoft didn’t make Server 2016 the final version Smile with tongue out )

image

I like the simplicity UI plus plethora of tools given for managing the servers.

image

This reminds me storage explorer tool. It’s very cool to manage server core systems just like you’re having full GUI on it Smile

In todays blog article I’ve only briefly touch the product capability. There are few more things to be discovered. I’m yet to explore the feature of Azure management from single console. Apart from that Microsoft promise we can manage our HYPER-V failover clusters and hyper-converged clusters as well. Apart from that if you’re using ASR to protect your hyper-v VM’s to Azure you can manage that fro Admin Center. As I said earlier I didn’t had all the lab scenarios in my hand right now but look forward to share my experience whenever I get my hands to such environments. In the meantime you’ll can share your exact experience when using this product.

PS: At this stage don’t take Windows Adin Center as you final destination to manage your Windows Server environment. At this stage Microsoft is very clear Admin Center will not replace your MMC consoles, Monitoring tools like SCOM, OMS and rest management tools. But Microsoft has given very clear picture where they heading on server management path. It would be really amazing when this product improved to manage services on server less environments as well.

Azure Site Recovery–Story revamped using new portal

In this blog post I’ll guide how to setup Azure Site Recovery (ASR) on the new portal using ARM model. If you’re not familiar with the ASR concept you can refer here. Compared to setting up ASR on old Azure portal, Microsoft ASR team carried out significant enhancement on the new portal and make it very much UI friendly.

In this blog post I’ll explain how to protect HYPER-V VM’s. You can protect VM’s hosted on single HYPER-V (Stand alone) or HYPER-V cluster (without VMM) using these steps. Few things I won’t cover in this blog post are how to create resource group, Virtual network….etc. I’ll provide relevant links for that for you to get in depth idea.

1. Wow to create a resource group in Azure – https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview

2. How to setup networking for ASR – https://azure.microsoft.com/en-us/blog/networking-infrastructure-setup-for-microsoft-azure-as-a-disaster-recovery-site/

So with the assumption you have HYPER-V server with bunch of VM’s (on-premise) and have a Azure tenant and in that you’ve created,

  • Resource Group
  • Created Virtual network
  • Created storage account to hold replicated VM’s data

Now let’s go ahead and create a Recovery Vault in the Resource Group you have created. In my case I’ve pre created a RG name as ASR-DR. Inside that I’m going to create the Recovery Vault name “ASR-RV”

image

image

Once the RV created we can follow the step-by step guide or based on your experience jump straight into the relevant steps. In below screenshot I’ve demonstrated the step by step method.

I’m selecting the option to protect the hyper-v vm’s which is not managed by VMM environment.

image

Now you need to create a “HYPER-V site” and then click on the “+ Hype-v server” and register the nodes. Once you complete that task of setting up agents into the on-premise server you’ll be registering your HYPER-V servers with the RV. In below picture you can see I’ve added two HYPER-V hosts.

image

in the next step you’ll need to define the Azure subscription. RV will read the resources in that vault and will highlight what is usable for ASR purpose.

PS: But I warn you to create the resources earlier for ASR purpose and not to borrow Smile

image

Now you need to define replication policy and associate. If you have done this step previously you only have to associate that, if not create a one. You can go ahead and create a new one keeping the defaults value and change them later.

image

Step 5 I’ve skipped that since I’ve make sure planning has been carried out previously.

image

Now the basic steps are completed and real game begins Smile

Go to “Replicate Application” section and start highlighting the VM’s you need to replicate to Azure for protection.

image

In the next step you need to map the Azure resources you created previously very carefully. I’ve highlighted the areas which need your special attention. Careful planning becomes a virtue in this scenario.

image

Now if everything goes smoothly you’ll be able to see the VM’s on the HYPER-V host server name list populated on Azure side. Go ahead and select the VM’s you need to protect,

image image

Finally you need to review the summary and approve to proceed for replication process to execute against the VM you select.

image

This will take little time to complete. After that for full sync will occur. For that time depend on your disk size and your internet connection speed Smile. I’m in the process of helping a client to upload over 2 TB data.

image

If you have very slow internet links (Like I’ve Smile) you can use Microsoft import/export method to export the VHD files to nearest Azure data-center via courier. Once Azure team upload your VHD to Azure storage account all you have to do is replicate the difference. Sounds easy? Well it is not! there are few steps you need to follow and it will cost you additional money but it all depend on the situation. You can find more information about it here.

My two cents advise is go ahead and setup the Recovery Vault and check the new options in the RV,

image

You’ll find new GUI and options given are so rich. In my future article I’ll cover more details about them and also the recovery procedure.

Error ID 70094: ASR Protection cannot be enable for HYPER-V VM

I came across above error when tried to setup ASR using new portal. Every step went find until I get below error,

image

clearly this highlight I cannot replicate the VM successfully to the Azure Recovery Vault. I’ve tried re-registering the Azure Site Recovery agent on the HYPER-V host as well. Though HYPER-V host register properly on the Recovery Vault VM protection fails with above error. On the hyper-v console I can see VM replication is on error state.

So finally meddle around the host logs I found out ASR has been setup previously and has not been removed properly. This means each VM replication also not completed and hanging around on error state. Only way to proceed is to clear those unsuccessful replication data on the host side targeting individual VM’s which is effected.

You need to run below mention PS command on each host targeting the effected VMs,

“$vmName = “<VM Name>”
  $hostName  = “<Host name>”
  $vm = Get-WmiObject -Namespace “root\virtualization\v2” -Query “Select * From Msvm_ComputerSystem Where ElementName = ‘$vmName'” -computername $hostName
  $replicationService = Get-WmiObject -Namespace “root\virtualization\v2”  -Query “Select * From Msvm_ReplicationService”  -computername $hostName
  $replicationService.RemoveReplicationRelationship($vm.__PATH)“

PS: Replace the <VM Name> with your effected VM name, <Host name> with your HYPER-V server name and run on the HYPER-V host.

Once that completed go ahead and try enabling replication for each VM from Azure console side.

PS: If you need to know about how to setup ASR on the new portal you’re in luck. Stay tune for next blog article Smile

How to encrypt disks on Azure VM’s

“Information protection” no wonder this word has been making big buzz around the world regardless of the business size. We have seen major cyber attacks, malware attacks which even cripple the Enterprise companies finically and reputation wise. So in this article I’m looking at one area of prevention solution offered by Microsoft team long time back. Now it’s extended to Microsoft Azure VM’s as well. Disk encryption is not a new term, we always had heard under Information Security practices consultants highlight how vital to back the data and keep them offshore. Same time they request this data to be encrypted in case fall into wrong hand.

But have you thought about how to protect running VM’s in your data-center or on Azure? Actually there are couple of ways you can approach or that. I recommend all of them in phase method based on your budget and time.

Antimalware
Compliance
Hardware Security Module (HSM)
Virtual machine disk encryption
Virtual machine backup
Azure Site Recovery
Security policy management and reporting

List can be going on over the time with new addons Smile. In this article I’ll describe how we can protect virtual machines using disk encryption technology. If you’re a HYPER-V fan then read about Shielded VM’s as an additional information.

Ok back to the main topic. This technology is referred as Azure Disk encryption which leverage Microsoft Bitlocker disk encryption. (I do hope now it makes sense to you all). Azure supports encrypting Windows VM’s using Bitlocker technology as well as Linux VM’s using  dm-crypt feature which provides volume encryption for the OS and the data disks. All the disk encryption keys and secrets saved on Azure Vault on existing subscription. The data (or in our case VHD files) resides safely on the Azure storage. Read about Azure Key Vault technology here.

Disk encryption activity can be approached from several methods,

disk-encryption-fig1
Picture credits to the Azure team Smile

1. In case if you decided to upload a encrypted VM from your HYPER-V environment to Azure make sure to upload the VHD to storage account and copy the encryption key material to your key vault. Then, provide the encryption configuration to enable encryption on a new IaaS VM.
2. If you create the Azure VM from Azure marketplace template then just provide the encryption configuration to enable encryption on the IaaS VM.
3. In case if you’ve already created VM on subscription leveraging the Azure marketplace still you can follow the same steps thanks to Azure Security Center.

So let’s assume you already created the Azure VM using the marketplace and started using that for your requirement. Later stage you found out though Azure Security Center you’ve not followed the industry bet practices and it’s highlighting the potential security risk you’re exposed to. One scenario is disks are not encrypted!

image

As you can see I’ve 3 Azure hosted VM’s and they are having potential security issues and not enabling disk encryption is one of them. On this article I’ll focus on one VM (VM01) which is running server 2012 R2 enabling the disk encryption.

First things first you need to get Azure PowerShell modules setup to your desktop / laptop. You can download them from the Azure download page.

image

After that you’ll need to get a PowerShell script to do the job. You can get that script from here. Copy the script and save it with any name you prefer. Make sure it’s extension as PS1.

Now you need to open the script using PowerShell ISE.

image

When you run the script you need to provide following information (orderly manner)

Resource Group Name – This is the RG name where you’ve hosted your VMs

Key Vault Name – Place where your keys will be saved and protected. During the execution of the script it’ll ask for a Key vault. If you didn’t have one create just proceed and it will create a key vault automatically.

Location – Where you Resource Group location. In my scenario it would be “southeastasia”
Tip: notice there are no space between the name. This is very important to remember.

Azure Active Directory Application Name – This is for the Azure Active Directory application that will be used to write secrets to the Key Vault. If you haven’t created one script will create one for you.

Now you’re aware the information you need to provide. Let’s proceed with the execution of the script under PowerShell ISE

image

If you get above screen that mean phase 1 activity is completed Smile 

Now it’s time to get ready to target a VM and encrypt the disks. For this part you need to tell PowerShell which VM you’re targeting. In the PowerShell type below command

$vmName = “<VM name>”

Replace <VM Name> with your VM hosted in that resource group. In my case it’s $vmName = “VM01”

Now in the above PowerShell script line 185 highlight the command to encrypt the disks. Copy that and run it on the PowerShell window. Alternatively you can copy the command mentioned below.

Set-AzureRmVMDiskEncryptionExtension -ResourceGroupName $resourceGroupName -VMName $vmName -AadClientID $aadClientID -AadClientSecret $aadClientSecret -DiskEncryptionKeyVaultUrl $diskEncryptionKeyVaultUrl -DiskEncryptionKeyVaultId $keyVaultResourceId -VolumeType All

If things go smoothly you’ll get below message on your PowerShell window,
image

This process will take around 10-15 min time to complete. On above screenshot you can see the command execution and result completion is successful.

After that you can return the VM properties and check the disk status. you can see below both OS and Data disks has been encrypted.

image

So any given time you add more VM’s to that resource group all you have to do is target the VM name and run the command line given above.

Note: Disk encryption on Azure is a really good option but need to be weighted carefully. If you want to backup the encrypted VM’s then encrypting need to be completed using KEK method. For more in-depth of Azure IaaS disk encryption refer to this article.

Azure Site Recovery (ASR) in action to protect Azure IaaS VMs

Update 04th July 2017 11:00 p.m.: Today Microsoft ASR team allow replicating Server 2016 VM’s  (Azure-to-Azure DR) scenario as well. These VM’s can support Storage space technology. Can check my short video here.

Kindly note this feature still in preview mode. Being said that I believe this is very important option for some customers. Based on customer feedback Microsoft has identified following points to justify this feature.

  • You need to meet compliance guidelines for specific apps and workloads that require a business continuity and disaster recovery (BCDR) strategy.
  • You want the ability to protect and recover Azure VMs based on your business decisions, and not only based on inbuilt Azure functionality.
  • You need to test failover and recovery in accordance with your business and compliance needs, with no impact on production.
  • You need to fail over to the recovery region in the event of a disaster and fail back to the original source region seamlessly.

So being said that below are my observations on ASR for Azure IaaS VM’s.

  • Setup and configuration is very much easy (Of course careful planning is required)
  • VM’s with Managed disks are not supported (This option will be coming soon)
  • You Site Recovery Resource Group has to be created on different region and cannot be on the same region where you production VM’s exists.
  • Automated replication. Site Recovery provides automated continuous replication. Failover and failback can be triggered with a single click via GUI.
  • Minimum replication time interval is 5 min (Wish this will be improved soon)
  • Just like protecting and testing on-premise VM’s to Azure, you can run disaster-recovery drills with on-demand test failovers, as and when needed, without affecting your production workloads or ongoing replication.
  • You can use recovery plans to orchestrate failover and failback of the entire application running on multiple VMs. This can be controlled via runbooks (very nice feature)

Ok now let’s get back to action Smile

To make things easier I’ve went ahead and created two RG (Resource Groups) in advance in two regions. I hope name convention is easy to understand it’s purpose.

image

Inside the ASR-PROD I already created single Server 2012 R2 VM.

image

So now we have a production VM ready to b protected. Next step is to create Recovery Vault on destination RG.

image

image

Select the VMs you want to replicate, and then click OK.

image

if you want you can override the default target settings and specify the settings you like by clicking Customize.

image

Once given command to execute Azure Recovery service will go ahead and do the job Smile

image

Initial replication might take some time. It all depend on how many number of disk you have in your Iaas VM and their size. But I am pretty sure it’s lot faster than uploading your on-premise datacenter VM to Azure scenario. I have experience 3-4 days to upload single VM to Azure Smile

Finally the success results would be as follows,

image

Nice GUI work from Azure ASR team visually showing which to which region VM getting replicated to,

image

Experience the DR drill. For this under the Site Recovery click the “Test Failover” option. This will create VM on the ASR RG. Once the test is complete you can select the option called “Cleanup test failover” This will delete the VMs that were created during the test failover

image

Tips:

During my demo lab creation came-up with below mentioned error. Problem is newly added disk is not be initialized inside the guest OS. Due to that reason ASR unable to replicate that disk to DR site.

image