New EUT Process Stops Helpdesk Tickets in Their Tracks

End users now have a critical role to play in validating new features and functionalities in production rather than in the development environment thanks to the success of a new testing process.

The End User Testing (EUT) process, which kicked off in January, enlists end users for the first time in this process on the 4 Innovation (4i) and Accountable Care Organization Management system (ACO-MS). 

CMS uses the 4i system to manage Alternative Payment Model participation, and the ACO-MS is used by accountable care organizations participating in the Medicare Shared Savings Program to manage their agreements, participants, contacts and other operational details. 

The EUT process is designed to proactively detect performance issues so they can be resolved before the new feature or functionality is widely used and triggers helpdesk tickets.  

“This process started out as an enhancement to our existing process because no performance testing data was available in the production environment,” says Karan Patel, an IT Specialist in OIT's Enterprise System Solutions Group (ESSG). "So we said why not have those who will be using the releases on the participating systems on a daily basis do the testing."

Individual testing sessions are conducted monthly. They usually last about 20-30 minutes depending on the number of screens and steps that users are being asked to validate, the feature/functionality complexity of the release, and participant questions. 

“The EUT sessions we’ve conducted so far have been a tremendous success even in the short time we have been conducting them,” Patel says. 

The first sessions in January identified a gate window issue that occurred because of improper configurations on the back end. If the EUT sessions weren’t held, Patel adds, this issue would have triggered numerous helpdesk tickets along with frustrations from our end users.

“Lastly, if there is still time left, we would ask the end user to let us know what their thoughts are regarding the system,” Patel says. “We also capture what the user likes and what the pain points are when using the system(s).”

Patel adds that end user suggestions will be taken back to the development team for further discussion. 

Since the initial results in January, no post-release issues have been identified during EUT sessions, which Patel considers a “really good sign.”  

Currently, EUT is being incorporated into the existing product release process for both the Mi and ACO-MS systems. Patel says future testing sessions will be divided into two parts – actual testing/validation and feedback from end users about the tested feature or functionality.

The EUT process was established by Patel’s predecessor, Kelly Gelletly, who is now Deputy Director of the Division of Enterprise Services. 

For more information on the EUT process, contact Karan Patel at Karan.Patel@cms.hhs.gov.