Artificial intelligence and machine learning technologies have been increasingly embedded in software development for several years. For scalable microservices architectures, these technologies enable automation of complex decision-making processes, improve resource efficiency, and unlock intelligent service delivery at scale.
However, moving from experimentation to real-world deployment remains challenging for many organizations. It requires a clear strategy for packaging, scaling, securing, and monitoring ML workloads within distributed systems.
Serhii Makhnovskyi, a certified senior DevOps and Software engineer with over 10 years of experience and a jury member of various IT competitions (International IT-Universe, etc), shared practical strategies for bridging the gap between data science and production environments. He emphasizes that machine learning improves, rather than disrupts, existing DevOps processes, and proves this with his own example.
Understanding the Gap between ML and Microservices
Taking machine learning models from the experimental domain to production-ready microservices is a complex task. However, the demand for this process is constantly growing. According to data published in IJIRSET, the microservices architecture market has a CAGR of 18.5% from 2024 to 2032.
As Serhii Makhnovskyi coordinated one of the competitions of the international IT-Universe competition, he often observed the difficulties young developers face. Most often, there are gaps between creating models and their subsequent deployment to production. Platforms such as Jupyter Notebooks or Google Colab are usually connected for development, and they work well for exploratory analysis and prototyping. However, in a production environment, they are significantly inferior in terms of reliability.
The point is that production environments require a more comprehensive approach. First, ML models should be accessible via APIs for seamless integration with other services. Secondly, in a production environment, constant version control is important to keep track of iterations, updates, and rollbacks. Finally, the usual platforms of Jupyter Notebooks and Google Colab lack scalability, especially when failures occur and graceful recovery is required.
To bridge this gap, it is important to adopt MLOps practices. MLOps combines machine learning with DevOps principles to optimize the deployment, monitoring, and management of machine learning models in production. By implementing MLOps, organizations can achieve greater accuracy, reliability, and, most importantly, support for real-world apps from their models.
Packaging ML Models for Microservices
To properly deploy ML models in microservices architectures, Serhii Makhnovskyi recommends paying attention to several important points:
1. Containerization of models with Docker and serving frameworks.
This makes it much easier to deploy uniformly across environments. Developers should also be aware of the different frameworks and tools that optimize the process: Flask and FastAPI are suitable for serving models as APIs; TensorFlow Serving and TorchServe can serve TensorFlow and PyTorch models, respectively, providing version control and model logging.
2. Understanding model serialization formats.
The appropriate serialization format will affect compatibility and performance, so it is worth understanding them:
- .pkl (Pickle): Typically used with scikit-learn models, but is Python-specific and can pose a security risk if not used properly.
- .pt / .pth: Native formats for PyTorch models, suitable for storing model weights and architectures.
- .h5: When working with Keras models, the format lets you store the optimizer state, weights, and model architecture in a single file.
- .onnx: Provides interoperability between different deep learning frameworks.
3. Storing models using registries and object storage.
Serhii Makhnovskyi notes that at this stage, it is important to choose the right object storage solutions: MLflow (a centralized repository for managing the ML lifecycle), services for storing large artifacts—Amazon S3, Google Cloud Storage (GCS), and Azure Blob Storage. By the way, MLflow can be configured to register artifacts directly in these storage solutions, simplifying integration.
Deploying ML Services to Kubernetes
Kubernetes can also be used to simplify the launch of ML models. The platform services provide various options that developers can use according to the following algorithm:
1. Serverless inference with Kubernetes and Knative.
Knative adds serverless features—it can automatically scale model modules up or down (even to zero), saving resources and costs.
2. CPU, GPU resource management, and autoscaling.
As Serhii Makhnovskyi recalls, students often forgot about CPU and memory limitations during training and completing Olympiad tasks, which is why models could empty the entire cluster. It is important to pay attention to this, and you should also use node selectors to run GPU-intensive models only on compatible hardware.
In addition, if you add HPA modules, you can configure a smooth increase/decrease in services based on actual usage.
3. Connecting machine learning services to other microservices.
Another advantage of Kubernetes is stable names and IP addresses. The ML API will always be available when using the platform. DNS resolution also simplifies the task. You can use ml-service.default.svc.cluster.local to call the service. A Service Mesh, such as Istio, can help with advanced needs: traffic management, encryption, and monitoring between services.
Thus, with Kubernetes, deploying ML models will be more flexible and resource-intensive.
Security and Secrets Management
Finally, when integrating ML into a microservices architecture, it's also important to keep data security in mind—API tokens, encryption keys, and credentials. To ensure a smooth process, Serhii Makhnovskyi recommends following these steps:
1. Handling sensitive data.
While Kubernetes secrets are designed to store tokens and passwords, they are not encrypted by default—only base64 encoded. You can configure security by enabling encryption at rest and using RBAC to restrict access to secrets. The senior developer also warns that credentials should never be hardcoded—environment variables should be used.
2. Using HashiCorp Vault for reliability.
Kubernetes secrets are simple and suitable for basic needs, but they lack robust access controls or audit logs. HashiCorp Vault is better suited for production: it creates dynamic secrets, logs all access to them, and enforces fine-grained access policies. That being said, HashiCorp Vault works seamlessly with Kubernetes to automatically inject secrets into pods.
3. Securing access to machine learning services.
Finally, there are a number of things to consider. First, use network policies to manage communication. Next, add TLS encryption to secure data in transit and use JWT/similar for authentication. Finally, consider using an API gateway to secure and monitor model endpoints.
Conclusion
Integrating ML models into microservices architectures is now critical to scaling AI in applications. As Serhii Makhnovskyi has learned from working with and mentoring students, this process can be complex and time-consuming at every stage. However, structured packaging, secure deployment, and consistent monitoring make the process easier.
By applying DevOps principles to machine learning workflows—MLOps—teams can bridge the gap between experimentation and production. Tools like Docker, MLflow, Knative, and Kubernetes provide the technical foundation, while resource management, secrets handling, and API-level integration ensure that these models are performant, secure, and maintainable.
Scalable machine learning does not exist in isolation. It must be integrated into a broader system supported by strong engineering processes. With the right integration strategy, DevOps tools and machine learning can leverage each other.
