Q&A: IDR Data Migration with Murari Selvakesavan
PlanetOIT recently asked Murari Selvakesavan, Director of OIT's Division of Enterprise Information Management Services, to share his insights on the IDR migration and what it means for CMS going forward.
Q: Why did CMS decide to migrate IDR to the cloud and what does it resolve?
A: The migration is a necessity. We started in September 2019, and the main business need is that we are on-prem in a fixed capacity right now, which means our storage and compute are all bundled together in a vendor appliance. If we have to increase capacity, it’s not easy. It is a contractor-owned, contractor-operated infrastructure arrangement. The current contracting structure doesn’t pan out well for adding more capacity.
In the past we had to increase capacity for storage more so than compute and had to keep bringing in new appliances to expand, which was costly and expensive. Plus, there’s a lot of coordination we had to do because we have over 4,000 users and 20-plus different application teams that use the IDR as a source system to conduct their program level operations. It’s not like we can switch out the device over the weekend. It doesn’t happen like that.
When we upgraded infrastructure, there were a lot of hours spent on coordination. If you had to bring new equipment into the Baltimore Data Center, the planning had to start a year earlier. That was a big pain point for us.
Q: What prompted OIT to consider a cloud migration for the IDR?
A: We had been living with the legacy system for years and when the cloud really picked up speed, we started looking at what we could do. One of the options was to modernize the IDR. We tested out some alternative platforms, and then landed on Snowflake because of its cloud native architecture – it’s built for the cloud ground up.
The main essence of that is the elasticity and separation of storage and compute, which are a significant necessity for us to have. We can scale up and scale out compute capacity however much we need without impacting the storage or vice versa. After a successful pilot effort with the Snowflake SaaS platform, we started our migration journey in September 2019. This is now the fourth year of the migration.
Q: What other considerations were taken into account before the decision to migrate was made?
A: We had to make sure all the impacted systems and applications and all the users were also migrating along. This was a big focus for us because the real adoption of the cloud system comes from the users of that environment. They must embrace it. So, a lot of that coordination had to be done here.
Q: What are the big benefits of migrating to the cloud?
A: Customers have the flexibility to use the IDR Cloud to meet their needs, without worrying about any physical hardware restrictions. The IDR cloud offers centralized storage that scale independently from compute, cost transparency for customers based on their compute usage, and a flexible ecosystem that allows us to onboard various toolsets as needed. If workloads need additional compute, we can provision that almost instantaneously without impacting anyone. All the cloud advantages that you would expect in a typical cloud deployment apply here as well.
Q: In terms of data usage, what kind of time savings can users expect in the cloud environment?
A: The beauty of being in the cloud and the advantage of Snowflake SaaS platform is, you don’t have to deploy or install any infrastructure. It’s all managed by Snowflake within the FedRAMP authorized environment that the IDR Cloud resides in. We don’t have to use human capital to manage the infrastructure.
With the on-prem environment we have a lot of DBAs and system engineers to maintain and administer the data platform Now that doesn’t have to happen for the IDR Cloud, as the database is hosted in the Snowflake vendor managed SaaS platform. From an infrastructure maintenance perspective there’s not much of a need for us to interface with that technology. The one-year planning and six months of coordination for infrastructure upgrades — none of that exists in the IDR Cloud. Things get done over the weekend. The vendor publishes it and it’s available across the board.
Q: How did the migration roll out?
A: To keep it simple, we rolled out the migration in two phases. Phase 1 was completed in the summer of 2022. It was basically establishing an IDR footprint in the cloud in Snowflake and getting all the historical data from on-prem to the cloud, and then redesigning all the data engineering processes to ingest and keep the data current and in synch between the IDR on-premises and cloud systems. Those ingestion processes have all been set up in the cloud as part of Phase 1.
We had about 600 terabytes worth of historical data that we migrated. Currently, we have two systems running in parallel – the IDR Cloud in Snowflake and the legacy IDR Teradata. Technically, users can go to each of these environments and run queries and get the same results.
Q: How long do you expect to maintain the two environments?
A: They will co-exist until November 2023. We’re looking to shut down the on-prem IDR by December 2023. That’s our goal, which also aligns with the OIT/IUSG’s timeline to shut down the CMS Baltimore Data Center (BDC) after migrating everything to the Equinix Data Center in Ashburn, Virginia, by the end of 2023. There are multiple contractual implications if this timeline is not met.
Q: What do you want to achieve in Phase 2 of the migration in 2023?
A: We want to finish up Phase 2 of the cloud migration, shut down IDR on-prem, and develop a user-friendly IDR Cloud Data Catalog that shows the end-to-end data lineage, organizes data assets for ease of access and consumption by users, and has the capability to integrate with the OIT Enterprise Data Catalog that is currently being conceived.
For more information on the IDR, email CMSIDR_Support@cms.hhs.gov.