Imagine your organization encounters a project where you have to switch storage vendors... What would you do? To begin with, you will need to evaluate and test the performance of the storage providers on your servers. At Civo, we faced a similar project, allowing us to test several storage providers on our bare metal servers. In this article, we will discuss what you should look for while migrating to a different provider and the ways you can test these providers.
Who is Civo?
Civo is a cloud-native service provider enabling users to host core applications with ease. With the click of a button, Civo helps to speed up development, increase productivity and reduce costs. Part of this deployment includes getting customer data and bare operating systems installed in the cluster. With this, Civo currently hosts over 200 terabytes of customer data over three geographical regions.
Why do we need to migrate data?
Many reasons can bring the need to migrate data from one cloud storage provider to another. This may include cost-effectiveness, trying out other providers in the market, terminating contracts with a previous provider, and so on. As for Civo, our contract was up for renewal. Although the previous provider was performing well, it gave us an opportunity to explore different projects and companies in the Kubernetes storage space that fit our requirements.
Dinesh Majrekar, CTO at Civo speaks more about this topic and the steps required to evaluate the market, testing vendor claims in the real world and what needs to be considered outside of the purely technological aspects of any product selection.
What to look for when searching for storage providers?
There are several things to look for while looking for a storage provider. Some of them are listed below:
Partnership
The partnership is one of the most important things to look for while looking for a storage provider. The partnership between your company and the storage vendor will mean that you can grow together. In addition, the partnership will not only get you a good storage experience with the product, but you will also get support when things go wrong.
Enterprise support options
Another important thing you should consider while choosing any storage vendor is their ability to provide enterprise support options. You need to know how you are getting support if things go wrong. Support can be provided through various means such as slack channels, GitHub issues, and enterprise support mechanisms where you have to get through different tiers of support to solve your problem. You should ensure to get a fix in place with a long-term solution.
Open-source and avoid vendor lock-in
One thing you should keep in mind while choosing the storage vendor is that the product should be as open-source as possible. Mark Boost, CEO at Civo recently spoke more about open-source in OpenUK’s latest report. Along with this, avoiding vendor lock-in must be a key component. We can term vendor lock-in as a phenomenon where you are too dependent on the product and you are forced to continue using it regardless of the quality over the time.
Underlying technology
The storage and landscape environment is constantly changing, and as a result, you need to know what technology vendors are using under the hood. You need to see if you agree with the technology being used by the vendors and whether other options are available. Based on these factors, you need to do an evaluation to determine if this is the correct storage vendor for your organization.
Performance with scalability and durability
During selection, you should ensure that your chosen solution should be performant, scale according to the needs, and have data durability. These should be taken as guaranteed as the vendor you will partner with should have done the performance, scalability, and durability tests.
How did we test the options?
At Civo, we are lucky to have a representative test environment which is a replica of what we deploy to production. As a result, we were able to install the vendor-provided storage options and run our installation stack against it. With that, we compared the speed of the launches between the current provider and the new providers, with our key metric being cluster launches. For example, if a storage solution launches a cluster 10 seconds slower than our usual launch time of 90 seconds, you can feel the buildup. For traditional use cases, such as launching a database, you will not be able to feel the buildup because it will run for an extended period of time and will not care about the launch time and the initial provision time.
The next thing we did while testing was to evaluate the performance loss against the raw disk access. It measured the overhead that storage vendors have while writing data to disk or retrieving data from a remote host. For this part of the testing, the main element that we cared about was the percentage loss compared to the direct access of the disk.
As we have a relatively small team, built-in monitoring and rolling updates are essential from the operations perspective. For some solutions, you need to shut down all workloads to do an upgrade. An example of this is that Civo currently has over 2500 VMs deployed, and to shut all of them down because of a storage upgrade is a big operation. This is why a rolling update is a must!
Also, as we looked for an enterprise solution, it came with a license agreement and enterprise support. We made sure that the commercials were stacked up. Especially as the product we offer is of low margin, the commercials are important.
What are the tools used in the process?
At Civo, a pipeline already exists where we use Terraform and terragrunt to deploy out to all of our regions. During testing, we needed to ensure that each vendor could fit into our pipelines by not adding more overhead to our deployment processes. Automation is really vital in this scenario as we plan to scale further in the future.
For testing, we have used ansible so that we could get repeatable tests. We don’t run workloads for things such as overall disk access and performance testing daily; hence, we have used ansible to test that. With this, we could put a repeatable guaranteed test against each storage solution as they were installed.
We’ve also used FIO, a standard disk and performance check tool. This gives a report of the raw disk access for reading, writing, sequential reading and writing, and random reading and writing. In addition, it helps us test and compare the performance across the different vendors. Finally, it helps us test the performance across the replications where you can see the change in the performance profile if you have two to three copies of the data.
Another tool we used during testing is KubeVirt, through which we tested the integration between KubeVirt and each vendor. If you want to learn more about KubeVirt, read our tutorial by Saiyam Pathak here.
Conclusion
There are a lot of storage providers out in the market. To choose the right one, you need to evaluate them according to your needs with the help of the right tools and processes. An open-source, performant, scaled, and durable solution with enterprise support that can easily integrate with your stack and pass the tests without any errors can be perfect.
If you are interested in learning more about our cluster, you can get hands-on by launching a cluster in seconds through our k3s-powered Kubernetes service. Also, if you are new to Kubernetes, you can learn everything you need to know to get started with Kubernetes with Civo Academy!
