Compliance & audit services
Speed processing of large models
NCS Analytics is a leader in providing dynamic, comprehensive, data-driven solutions for governments and financial institutions as they interact with traditionally high-risk industries. The NCS team and their patent-pending analytics utilize a multitude of data sources, providing holistic insights to clients about the markets in which they work.
“When you have an industry that is highly taxed or highly regulated, you’ll have someone who is incentivized to cheat the system,” notes Adam Crabtree, CEO of Denver-based NCS Analytics. “We help ensure that these business communities are able to sustain their operations legally and that financial institutions can serve them with confidence.”
The insights NCS delivers come in the form of highly detailed alerts that NCS customers can use to identify suspicious behaviors of various market participants. For example, an alert may pinpoint a cannabis grow operation whose reported yields don’t match standard expectations for the variety they’re growing – something a regulator may want to follow up on. Through the NCS Platform, customers can see the specifics around each alert, along with questions intended to help facilitate meaningful conversations with licensees.
The NCS Analytics team creates complex models that ingest data from an array of sources including socioeconomic, bank, tax, utility, licensing, seed-to-sale tracking and point of sale data. Tapping the company’s advanced analytics engine, analysts are able to identify outliers. They use this information to flag or alert regulators on transactions or behaviors that may need to be reviewed further. What results is a dynamic picture of a marketplace.
“Our customers are looking for guidance,” says Crabtree. “In order to deliver that, we have to build models that can handle the vast amounts of data being ingested. Implementing efficient tools is crucial in order to provide actionable intelligence.”
Before utilizing TidalScale, the NCS analytical processes took a significant amount of time and effort to complete due to traditional computing systems and their limited abilities to handle the large quantities of data being ingested.
“The data grows daily, and at very large incremental rates,” says Adam Zukowski, a data analyst at NCS. “Every time we onboard a new customer, the data grows rapidly.”
Because detecting outliers requires intensive statistical computation, Zukowski says, the task proved too much for NCS’s existing resources—a combination of local systems and cloud servers leased from Amazon Web Services.
At times, the problem was crippling. The models became so large that some took 30 days to fully process. As a workaround, the team would parse the full data sets into more manageable workloads—a necessary, but not ideal, approach.
“We would load a small sample of a few tens of millions of data points onto our local machines, start a model training at the end of the day, and hope our local machines had not crashed when we came in the next morning,” recalls Isaac Freitas, an NCS senior data analyst. “We tried some processing and aggregating in the cloud to speed things up, but we were still left with limits.”
All this changed when the company implemented Software-Defined Server solutions from TidalScale. TidalScale software combines the resources of multiple commodity servers into a single system to provide as much memory, cores and I/O as organizations need to handle even their largest workloads.
With a Software-Defined Server, what once took 30 days now takes just three hours.
“With TidalScale, we can pull the entire dataset into memory and run multiple models simultaneously in a matter of hours instead of days,” explains Freitas. “Now with even more data flowing in from clients and partners, we can handle each data set with confidence, knowing our infrastructure can handle the load.”
Freitas says NCS’s entire development team now can process full-size models in parallel on the same Software-Defined Server. “This would have been impossible without TidalScale,” he says.
To improve the accuracy of the analytics that NCS bases its models on, developers train over several iterations. Zukowski says the team can iterate models 10 times or more a day, further accelerating the pace at which NCS reaches a model that’s ready to create final production output for clients.
“Using TidalScale has been a game-changer,” adds Crabtree. “Between our proprietary analytics and our ability to train models and crunch the numbers rapidly, we are in a very unique spot.”
By iterating efficiently, the development team has more time to be creative. “We can throw out tons of graphics based on data, very quickly and repeatedly, whereas previously we might have been able to do one,” says Zukowski. “We can test various mapping scenarios, like scatter plots, 3D plots, or interactive 4D plots. It allows us to have fun and see where the data leads us—without having to wait a week just to get through a basic part of what we do.”
Crabtree says the productivity gains that have come from using TidalScale Software-Defined Servers have kept operational costs low.
“We’d need at least three times as many analysts, three times the workstations, and three times the overhead allocations (such as space and benefits) to get done on traditional systems what we can achieve with our current team on TidalScale,” says Crabtree. “And we’d dramatically increase our runtime costs, since AWS charges by the server and the power being consumed.”
For Crabtree, the ability of TidalScale Software-Defined Servers to easily handle any model—while knowing his team can reconfigure their virtual server in minutes if their workload changes—is a strategic edge for NCS.
“TidalScale gives us a competitive advantage,” he says. “Because we can run our predictive analytics engine entirely in memory, other companies just can’t compete with our analytics.”
For Zukowski, being able to work on a system of virtually any size also comes with bragging rights: “I have developer friends who work at other places,” he says. “They don’t have access to the computing power that we do. When they learn what resources we have, their jaws drop.”