Open source refers to non-proprietary software that allows anyone to modify, enhance, or view the source code behind it. Our resources enable programmers to work or collaborate on projects created by different teams, companies, and organizations.
There is a conventional approach to learning a framework such as Flask: learn with a tutorial, then build something small, and gradually increase functionality. There are dozens to hundreds of such tutorials, and they are very helpful. Here we offer a complementary approach, one that entirely reverses the script. Build a complete running project you can explore within a minute, then learn how to alter it, debug it - and then how to create it, literally in seconds. App Fiddle: An In-Action Flask Tutorial Tools like JSFiddle are extremely useful. Without installation, you can use your browser to explore existing JavaScript/HTML code, alter it, and see the results. Here, we apply this approach to an entire app: an App (Flask) Fiddle (link at the end). Like a JSFiddle, it opens in your browser; no install. But it's a complete Flask app: a running project, with a database, accessed with SQLAlchemy Accessed via VSCode, running in your browser, courtesy of Codespaces Codespaces is a remarkable new product from GitHub. When you click the link at the end, it requisitions a server, installs your project (and all its dependencies, such as Python and Flask), and opens it in VSCode in your browser. You can also use this App Fiddle to explore Codespaces, how to set up a dev container, and use it on your own projects. The link (at the end) actually opens 3 projects. The first is a minimal Flask/SQLAlchemy app. It has a README: use it to explore the code, run it, alter/debug it, etc. Deliver While Learning But that's not all. While the first project shows it's pretty simple to create a single endpoint, gather some data, and return it, it's a lot more work to create an entire project (multiple endpoints, an Admin App, etc). That's a horse of an entirely different feather! So, we've created API Logic Server. It's an open-source Python app, already loaded into our Codespace project. It creates an entire Flask project with a single command, like this: ApiLogicServer create --project_name=ApiLogicProject --db_url=nw- This reads your database schema (here, a version of Northwind) and creates a complete, executable project, instantly: API: An endpoint for each table, with filtering, sorting, pagination, and related data access. Swagger is automatic. Admin UI: Multi-page/Multi-table apps, with page navigations, automatic joins, and declarative hide/show. It executes a YAML file, so basic customizations do not require HTML or JavaScript background. Custom UIs can be built using your tool of choice (React, Angular, etc), using the API. Fully Customizable: Standard Python, Flask, SQLAlchemy Creating the executable project requires no background in Flask, SQLAlchemy, or even Python. In fact, you can use the created project to learn these technologies, by "fiddling" with a running system that's already delivering value (e.g, enabling custom UI dev, integration, etc). That's because the created project is a standard Flask/SQLAlchemy project. Customize and extend it with all the fundamentals you learned in conventional tutorials, and in the App Fiddle, with your favorite IDE. Unique Spreadsheet-Like Business Rules As an experienced app developer, I think of projects as about half the back end and half the front end. Your mileage may vary, but the backend is certainly a lot of work. Multi-table derivations and constraints applied on update: E.g., the customer's balance - the sum of the unshipped order amounts - cannot exceed the credit limit Authorization and authentication E.g., users must enter a valid id and password to gain access Their roles determine what database rows they see (e.g., a multi-tenant application). API Logic Server enables you to declare spreadsheet-like rules to implement these. Rules are a very significant technology, but perhaps the most striking characteristic is that they are 40X more concise than code. These 5 rules represent the same logic as 200 lines of Python: Python Rule.constraint(validate=models.Customer, # logic design translates directly into rules as_condition=lambda row: row.Balance <= row.CreditLimit, error_msg="balance ({row.Balance}) exceeds credit ({row.CreditLimit})") Rule.sum(derive=models.Customer.Balance, # adjust iff AmountTotal or ShippedDate or CustomerID changes as_sum_of=models.Order.AmountTotal, where=lambda row: row.ShippedDate is None) # adjusts - *not* a sql select sum... Rule.sum(derive=models.Order.AmountTotal, # adjust iff Amount or OrderID changes as_sum_of=models.OrderDetail.Amount) Rule.formula(derive=models.OrderDetail.Amount, # compute price * qty as_expression=lambda row: row.UnitPrice * row.Quantity) Rule.copy(derive=models.OrderDetail.UnitPrice, # get Product Price (e,g., on insert, or ProductId change) from_parent=models.Product.UnitPrice) The third project in the fiddle illustrates both the rules and some "standard" Flask/SQLAlchemy customizations. A tutorial is included to help you explore these, run them, see how to debug them, etc. You can access the app fiddle by following the links at the top of this blog.
OpenSSH is a free and open-source suite of secure networking utilities that has become a critical tool for system administrators and developers who need to securely manage and access remote systems over unsecured networks. In this article, we will take a closer look at what OpenSSH is, how it works, and its importance in modern computing. OpenSSH History OpenSSH was developed in 1999 as an open-source implementation of the Secure Shell (SSH) protocol. The SSH protocol was developed as a replacement for the older Telnet protocol, which transmitted login credentials and other sensitive data in clear text over the network, making them vulnerable to interception and unauthorized access. OpenSSH was created by a team of developers led by Markus Friedl, and quickly gained popularity due to its ease of use, strong encryption, and ability to provide secure remote access and file transfer capabilities over unsecured networks. What Is OpenSSH? OpenSSH is a suite of secure networking utilities that provides encrypted communication channels between networked computers. It was developed as a replacement for the older, less secure SSH (Secure Shell) protocol, which was designed to provide secure remote login and command execution capabilities over unsecured networks. OpenSSH extends the functionality of SSH to include secure file transfer and other networking utilities. It was initially developed as a replacement for the older, less secure SSH (Secure Shell) protocol, and has become the de facto standard for remote access and administration of Linux and Unix systems. OpenSSH includes several tools, including SSH (Secure Shell) for secure remote login and command execution, SCP (Secure Copy Protocol) for secure file transfer, and SFTP (Secure FTP) for secure file transfer over an FTP-like interface. These tools are designed to provide encrypted communication channels between networked computers, protecting the confidentiality and integrity of data transmitted over the network. OpenSSH also provides a secure alternative to the Telnet and FTP protocols, which transmit login credentials and other sensitive data in clear text over the network, making them vulnerable to interception and unauthorized access. Overall, OpenSSH is a critical tool for system administrators and developers who need to securely manage and access remote systems over unsecured networks. How Does OpenSSH Work? OpenSSH uses a combination of symmetric and asymmetric encryption algorithms to protect the confidentiality and integrity of data transmitted over the network. When a user connects to a remote system using OpenSSH, the client and server negotiate a shared secret key that is used to encrypt all subsequent communication between them. OpenSSH also uses digital signatures to verify the authenticity of the remote system and the identity of the user. When a user logs in to a remote system using OpenSSH, the client and server exchange digital signatures to verify each other’s identity. OpenSSH works by providing a secure encrypted channel between two networked computers, allowing secure remote access and file transfer capabilities. When a user logs in to a remote system using SSH, OpenSSH first authenticates the user’s identity using a combination of username and password, or public and private key pairs. Once the user is authenticated, OpenSSH establishes an encrypted communication channel between the two computers. The encryption process used by OpenSSH involves generating a symmetric encryption key that is shared by both computers, and then using this key to encrypt all data transmitted over the network. The key is exchanged using a public key cryptography system, where each computer generates a public-private key pair. The public key is then exchanged between the two computers, and the private key is kept secret by each computer. Once the encrypted channel is established, the user can execute remote commands on the remote system, or transfer files securely between the two computers using SCP or SFTP. OpenSSH also provides several features to ensure the security of the encrypted communication channel. These features include: Host key verification: OpenSSH verifies the host key of the remote system to ensure the user is connecting to the intended system, and not to an impostor system. X11 forwarding: OpenSSH provides X11 forwarding capabilities, which allows a user to securely run X11 applications on the remote system and display them on the local system. Port forwarding: OpenSSH provides port forwarding capabilities, which allows a user to securely forward network traffic from one computer to another over the encrypted channel. Overall, OpenSSH is a powerful tool that provides a secure way to access and manage remote systems over unsecured networks. Its use of strong encryption and public-private key cryptography ensures sensitive data remains confidential and secure during transmission. Why Is OpenSSH Important? OpenSSH is a critical tool for system administrators and developers who need to securely manage and access remote systems over unsecured networks. OpenSSH is also an important tool for securing cloud computing environments: Secure remote access: OpenSSH provides a secure way to remotely access and manage systems over unsecured networks. It uses strong encryption to protect data transmitted over the network, ensuring that sensitive data remains confidential and secure. Secure file transfer: OpenSSH provides a secure way to transfer files between systems over unsecured networks. Its SCP and SFTP tools use strong encryption to protect data during transmission, ensuring files cannot be intercepted or accessed by unauthorized users. Protects against attacks: OpenSSH helps protect against attacks by providing a secure encrypted channel between networked computers. This makes it difficult for attackers to intercept or modify data transmitted over the network, reducing the risk of data theft or tampering. Open source: OpenSSH is an open-source tool, which means its source code is freely available to anyone who wants to use or modify it. This allows users to customize OpenSSH to meet their specific needs, and also ensures the tool is constantly being improved and updated by a large community of developers. Widely-used: OpenSSH is widely-used by system administrators and developers, and has become the de facto standard for remote access and administration of Linux and Unix systems. Its widespread use ensures it is well-tested and reliable, and that support and resources are readily available. OpenSSH Tools OpenSSH includes several powerful tools that are essential for secure remote access and file transfer. These tools include: SSH: This is the most widely used tool in the OpenSSH suite. It provides a secure remote shell, allowing users to log in to remote systems and execute commands as if they were sitting at the console. SCP: This tool provides secure file transfer capabilities, allowing users to copy files between systems over an encrypted channel. SFTP: This is a more advanced file transfer tool that provides an FTP-like interface for transferring files between systems over an encrypted channel. SSH-keygen: This tool is used to generate public and private key pairs for use with OpenSSH. Public keys are used to authenticate users and encrypt data, while private keys are used to decrypt data and sign messages. Overall, OpenSSH is a feature-rich tool that provides a secure and flexible way to manage remote systems and transfer files over unsecured networks. Its support for public-key authentication, X11 forwarding, port forwarding, and host key verification, as well as its support for multiple protocols, make it an essential tool for system administrators and developers. Benefits of OpenSSH OpenSSH provides several key benefits that make it an essential tool for system administrators and developers: Secure remote access: OpenSSH provides a secure remote shell, allowing users to log in to remote systems over an encrypted channel. This provides a secure way to access and manage remote systems, even over unsecured networks. Secure file transfer: OpenSSH provides secure file transfer capabilities, allowing users to copy files between systems over an encrypted channel. This protects data from interception and unauthorized access, ensuring sensitive data remains confidential. Strong encryption: OpenSSH uses strong encryption algorithms, including AES, to protect data transmitted over the network. This provides a high level of security, ensuring data cannot be intercepted or decrypted by unauthorized users. Open source: OpenSSH is an open-source tool, which means its source code is freely available to anyone who wants to use or modify it. This allows users to customize OpenSSH to meet their specific needs, and ensures the tool is constantly being improved and updated by a large community of developers. Security: OpenSSH is designed with security in mind and provides a secure way to remotely access and manage systems over unsecured networks. Its strong encryption and support for public-key authentication help protect against attacks and ensure sensitive data remains confidential and secure. Flexibility: OpenSSH is a flexible tool that can be used for a wide range of tasks, including remote access, file transfer, and port forwarding. Its support for multiple protocols and X11 forwarding capabilities make it a versatile tool that can be customized to meet a wide range of needs. Easy to use: OpenSSH is easy to use and does not require any special software or hardware to be installed on the client or server systems. It can be used from a variety of platforms and operating systems, making it accessible to a wide range of users. Reliable: OpenSSH is a reliable tool that has been extensively tested and is widely used by system administrators and developers. Its widespread use ensures it is well-tested and reliable, and that support and resources are readily available. Cost-effective: OpenSSH is a cost-effective tool that does not require any licensing fees or subscriptions. It is freely available and can be installed on any system, making it an affordable solution for managing remote systems and transferring files over unsecured networks. Conclusion OpenSSH is a powerful suite of secure networking utilities that has become an essential tool for system administrators and developers. Its strong encryption, secure remote access and file transfer capabilities, and open-source nature make it an ideal tool for protecting sensitive data and managing remote systems over unsecured networks.
Docker is a containerization technology that allows developers to package and deploy applications in lightweight, portable containers. These containers are isolated from the host operating system, which makes them portable across different environments and eliminates the “works on my machine” problem. Docker is a popular platform for creating and managing containerized applications; however, several alternatives for Docker can also be used for this purpose. Podman, Kubernetes, Openshift, LXD, Docker Swarm, BuidKit, and Mesos are some of the popular Docker alternatives available in the market today. In this article, we’ll discuss the following three Docker hub alternatives: Podman, Containerd, and LXD. So, let’s Begin! Podman Developed by RedHat, Podman is a daemonless, open-source, Linux-native container engine that is considered one of the best alternatives for Docker. Podman is used to build, run, and manage Linux OCI containers and container images. A container engine is an all-in-one software that is responsible for creating, running, and managing containers. A container engine provides an API or command-line interface for interacting with the containers, allowing developers to create, start, stop, and manage containers. Examples of container engines include Docker, Podman, and CRI-O. Podman uses the lib pod library, which provides a higher-level API for managing pods and containers. It also provides built-in support for rootless containers and improved security features. Advantages of Podman Easy to use: Podman has a simple and intuitive command-line interface that is similar to Docker’s command-line interface, making it easy for users who are already familiar with Docker to start using Podman. Compatible with Kubernetes: Podman can be used in confluence with Kubernetes, which means it can be used to run containers on a cluster adn locally. Support for multiple container formats: Podman supports both OCI and Docker container formats, which means it can run containers created using either format. Support for Cgroups v2: Podman supports Cgroups v2, which is a new version of the Linux kernel’s control group (cgroup) mechanism that provides more fine-grained control over resource allocation. Network support namespaces: Podman supports network namespaces, which allows you to use different network configurations for different containers. Differences Between Podman and Docker Docker and Podman are container engines, but there are some key differences between the two. Docker and Docker hub alternatives, such as Podman, are widely used and supported in the industry, and it depends on the specific use case and requirements of which one to use. Here are some of the key differences between Docker and Podman: Daemonless: Podman does not require a daemon to run containers, whereas Docker uses a daemon to manage containers. This means that Podman can run containers directly without the need for an additional service running in the background. Rootless: Podman can run containers without needing root access, whereas Docker requires root access to manage the container daemon. This makes Podman more secure, as it limits the potential attack surface. Image storage: Podman stores images in the local file system, whereas Docker uses a centralized image registry. This means that, with Podman, it is not necessary to have an internet connection to use local images. Networking: Docker uses its own networking stack, while Podman uses the host’s networking stack. CLI: Both have similar command line interfaces, so it’s easy to switch between them. Overall, Docker and Podman are powerful tools for containerization. For these two, and any other Docker alternatives, the ultimate choice between them often comes down to personal preference and specific use case requirements. Containerd Next on the list of Docker alternatives is Containerd. Containerd is a high-level, lightweight container runtime that provides a consistent and stable interface for running containers. Designed to be used as a daemon process that runs on a host system, it manages the lifecycle of containers by starting and stopping them, as well as providing other features, such as image management and storage. Containerd is also designed to work with other container orchestration tools, such as Kubernetes, to manage the scaling and scheduling of containers in a cluster. Advantages of Containerd Lightweight: Containerd is designed to be lightweight and fast, which means it has a small footprint and uses minimal resources. This makes it well-suited for use in high-performance and resource-constrained environments. Consistency: Containerd provides a consistent and stable interface for running containers, which makes it easier to manage and orchestrate them at scale. Flexibility: Containerd can be used with a variety of container orchestration tools, such as Kubernetes and Docker Swarm, which allows for greater flexibility in terms of how containers are managed and scaled. Plugins: Containerd has a modular design and support for plugins, which allows for easy customization and extension of its functionality. Security: Containerd provides a secure and isolated environment for running containers, and it has built-in support for image signing and validation. Support: Containerd is an open-source project with a large and active community, which means it has a wide range of support and resources available. Differences Between Containerd and Docker Containerd and Docker are container runtimes, but they have some key differences. Let’s take a look at these: Design: Containerd is designed to be a lightweight and minimal container runtime, while Docker is a more fully-featured container platform that includes additional components such as a built-in container registry and a management API. Functionality: Containerd focuses on providing a stable and consistent interface for running containers, while Docker provides a more comprehensive set of features such as image management and orchestration. Deployment: Containerd is intended to be used as a daemon process that runs on a host system, while Docker is typically deployed as a standalone service. Architecture: Containerd has a modular architecture that is designed to work with other container orchestration tools, while Docker has its own built-in orchestration features. Support: Containerd is an open-source project that is backed by a large and active community, while Docker is a commercial product that is supported by the company behind it. Plugins: Containerd has a pluggable architecture, which means it can be extended or customized using plugins, while Docker does not have a similar feature. Security: Containerd provides a secure and isolated environment for running containers and has built-in support for image signing and validation, while Docker does not have this feature by default. LXD Now, we’ll discuss one of the most commonly used alternatives of Docker in the list of Docker hub alternatives. LXD (Linux Containers Daemon) is a container hypervisor for Linux. It allows multiple isolated Linux systems (containers) to run on a single host, providing a lightweight alternative to virtual machines. LXD uses Linux kernel features, such as control groups and namespaces, to provide isolation, while also providing a simple and user-friendly command-line interface for managing containers. LXD is designed to work with existing Linux distributions and tools and supports a wide range of container images and formats, including Docker. It also provides advanced features like live migration, storage management, and network management. Developed and maintained by Canonicals, LXD is one of the well-known Docker hub alternatives and is the default container hypervisor for Ubuntu 20.04 and later versions. Advantages of LXD There are several advantages to using LXD as a container hypervisor. LXD is one of the most known Docker desktop alternatives available in the industry today. Take a look at the advantages of LXD below: Lightweight and fast: LXD uses Linux kernel features, such as control groups and namespaces, which are more lightweight and efficient than traditional virtualization methods. This results in faster startup times and lower resource overhead for containers. Easy to use: LXD provides a simple and user-friendly command-line interface for managing containers, making it easy to create, start, stop, and manage containers. Compatible with existing Linux distributions and tools: LXD is designed to work with existing Linux distributions and tools, and supports a wide range of container images and formats, including Docker. Advanced features: LXD provides advanced features, such as live migration, storage management, and network management, which allows you to move running containers between hosts without interruption, manage storage resources, and network interfaces within containers. Security: LXD uses AppArmor and Seccomp to provide additional security to the containers. Networking: LXD provides easy-to-use networking features to manage the container’s network interfaces, assign IP addresses, and create virtual networks. Scalability: LXD can run thousands of containers on a single host, making it highly scalable for large-scale deployments. High availability: LXD supports clustering features with HAproxy, allowing the creation of a highly available environment with automatic failover. Differences Between LXD and Docker LXD and Docker are containerization technologies, but they have some key differences. The decision of choosing Docker desktop alternatives should be made based on the use case and business requirements. Use case: LXD is a container hypervisor, while Docker is a container runtime. This means that LXD provides an additional layer of abstraction, allowing multiple isolated Linux systems (containers) to run on a single host, while Docker focuses on running individual containers. Containerization: LXD provides a more complete, system-level containerization experience, while Docker is more focused on application-level containerization. Design: LXD is designed to work with existing Linux distributions and tools and supports a wide range of container images and formats, including Docker. Docker, on the other hand, is focused on its own container format and ecosystem. Security Integration: LXD uses AppArmor and Seccomp to provide additional security to the containers, while Docker uses namespaces and control groups to isolate containers. Networking: LXD provides easy-to-use Networking features to manage the container's network interfaces and assign IP addresses, and create virtual networks, while Docker uses virtual networks based on IP addresses and network interfaces provided by the host. Overall, while Docker and LXD are powerful containerization technologies, they are designed to solve different problems and have different use cases. Depending on the use case, these alternatives of Docker can be used. How To Choose the Best Docker Alternative When choosing a Docker alternative, it is important to consider the following factors: Compatibility: Make sure the alternative is compatible with your existing infrastructure and technologies. Features: Evaluate the features offered by the alternative and see if they align with your needs. Support: Consider the level of support offered by the alternative and its community. Performance: Consider the performance of the alternative in terms of resource usage and scalability. Security: Evaluate the security features offered by the alternative and see if they meet your requirements. Cost: Consider the cost of using the alternative and compare it to other options. Conclusion So these were some of the popular alternatives for Docker. Each of these Docker alternatives has its own strengths and weaknesses, so it's important to analyze the pros and cons of each and study your business requirements before choosing any of these alternatives of Docker.
As with many engineering problems, there are many ways to build RESTful APIs. Most of the time, when building RESTful APIs, engineers prefer to use frameworks. API frameworks provide an excellent platform for building APIs with most of the components necessary straight out of the box. In this post, we will explore the 10 most popular REST API frameworks for building web APIs. These frameworks span multiple languages and varying levels of complexity and customization. First, let’s dig into some key factors in deciding which framework to begin building with. How To Pick an API Framework The first factor in choosing an API framework is usually deciding which language you want to work in. For many projects, depending on the organization you work with or your experience, choices may be limited. The usual recommendation is to go with a language you are already familiar with since learning a new language and a new framework can lead to less-than-optimal implementations. If you’re already familiar with the language, your main focus can be on understanding the framework and building efficiently. Once the language is decided, you may have multiple choices of frameworks that support your language of choice. At this point, you will need to decide based on what types of functionality you require from your APIs. Some frameworks will have plugins and dependencies that allow for easy integration with other platforms, some may support your use case more precisely, and others may be limited in functionality that you require, automatically disqualifying them. Making sure that your use case and functionalities are supported by the framework is key. Last but not least, you should also consider the learning curve and educational materials and docs available. As a developer, the availability of good documentation and examples are massive factors in how quickly you and your team can scale up your APIs. Before deciding on a framework, browse the documentation, and do a quick search to ensure that you can find examples that can guide you and inform you on how much effort is needed to build APIs in the framework of your choosing. Now that we have a few factors to consider let’s take a look at some popular framework options. Spring Boot Spring Boot is an open-source framework that helps developers build web and mobile apps. Developed by Pivotal Software, Spring Boot is a framework that’s intended to make the original Spring framework more user-friendly. You can easily start using Spring Boot out of the box without spending time configuring any of its libraries. Programming Language: Java Pros: - Quick to load due to enhanced memory allocation - Can be easily configured with XML configurations and annotations - Easy to run since it includes a built-in server Cons: - Not backward compatible with previous Spring projects and no tools to assist with migration - Binary size can be bloated from default dependencies To learn more about Spring Boot framework, you can check out the docs here Ruby on Rails Ruby on Rails was originally developed as an MVC framework, which gives it the name “the startup technology” among developers. The main purpose of the framework is to deliver apps with high performance. The high-performance standards of Ruby on Rails excited developers using Python and PHP, and many of its concepts are replicated in popular Python and PHP frameworks. Programming Language: Ruby Pros: Great framework for rapid development with minimal bugs Open-source with many tools and libraries available Modular design with efficient package management system Cons: Can be difficult to scale compared to other frameworks like Django and Express Limited multi-threading support for some libraries Documentation can be somewhat sparse, especially for 3rd party libraries To learn more about Ruby on Rails framework, you can check out the docs here Flask Flask is a Python framework developed by Armin Ronacher. Flask’s framework is more explicit than Django and is also easier to learn. Flask is based on the Web Server Gateway Interface toolkit and Jinja2 template engine. Programming Language: Python Pros: Built-in development server and fast debugger Integrated support for unit testing RESTful request dispatching WSGI 1.0 compliant Unicode base Cons: Included tools and extensions are lacking, and custom code is often required Security risks Larger implementations more complex to maintain To learn more about Flask framework, you can check out the docs here Django REST Django REST framework is a customizable toolkit that makes it easy to build APIs. It’s based on Danjgo’s class-based views, so it can be an excellent choice if you’re familiar with Django. Programming Language: Python Pros: The web browsable API is a huge win for web developers Developers can authenticate people on their web app with OAuth2. Provide both ORM and non-ORM serialization. Extensive Documentation Easy Deploy Cons: Learning Curve Does not cover Async Serializers are slow and impractical for JSON validation To learn more about Django REST framework. You can check out the docs here Express.Js Express.Js is an open-source framework for Node.js that simplifies the process of development by offering a set of useful tools, features, and plugins. Programming Language: Javascript Pros: Well Documented Scale application quickly Widely used and good community support Cons: Lack of Security Issues in the callbacks Request problems encountered with the middleware system To learn more about Express.Js framework, you can check out the docs here Fastify First created in 2016, Fastify is a web framework that is highly dedicated to providing the best developer experience possible. A powerful plugin architecture and minimal overhead also help make this framework a great choice for developers. Programming Language: Javascript Pros: Easy development Performant and highly scalable The low-overhead web framework that grounds this system minimizes operation costs for the entire application. Cons: Lack of Documentation and community support Not readily used in the industry To learn more about Fastify framework, you can check out the docs here Play Framework Play is a web application framework for creating modern, robust applications using Scala and Java. Based on Dynamic Types, Play integrates the components and APIs required for modern web application development. Programming Language: Java, Scala Pros: Intuitive User Interface Testing the Application simplified Faster development on multiple projects Cons: Steep Learning Curve Too many plug-ins which are unstable Maybe it doesn’t offer any features for backward compatibility. To learn more about Play framework, you can check out the docs here Gin Gin is a fast framework for building web applications and microservices in the programming language Go. It provides a martini-like API and enables users to build versatile and powerful applications with Go. It contains common functionalities used in web development frameworks such as routing, middleware support, rendering, etc. Programming Language: Golang Pros: Performance Easy to track HTTP method status code Easy JSON validation Crash-free Cons: Lack of documentation Syntax not concise To learn more about Gin framework, you can check out the docs here Phoenix Phoenix is written in Elixir and works to implement the MVC pattern. It will seem similar to frameworks like Ruby on Rails and Django. One interesting thing about Phoenix is that it has channels for real-time features which pre-compile templates. These templates work quickly, making the site smooth and easy to scroll through. Programming Language: Elixir Pros: Filters data that is safe and efficient Elixir runs on the Erland VM for improved web app performance. Concurrency Cons: Expensive Processing Speed Prior Erlang knowledge required To learn more about Phoenix framework, you can check out the docs here Fast API Fast API is a web framework for developing RESTful APIs in Python. It fully supports asynchronous programming, so it can run with product servers such as Uvicorn and Hypercorn. It has support for Fast API into popular IDEs, such as JetBrains PyCharm. Programming Language: Python Pros: High Performance Easy to Code with few bugs Short Development time Supports asynchronous programming Cons: Poor Request Validation Does not support singleton instances Main File is crowded To learn more about Fast API framework, you can check out the docs here Adding in API Analytics and Monetization Building an API is only the start. Once your API is built, you’ll want to make sure that you are monitoring and analyzing incoming traffic. By doing this, you can identify potential issues and security flaws and determine how your API is being used. These can all be crucial aspects in growing and supporting your APIs. As your API platform grows, you may be focused on API products. This is making the shift from simply building APIs into the domain of using the API as a business tool. Much like a more formal product, an API product needs to be managed and likely will be monetized. Building revenue from your APIs can be a great way to expand your business’s bottom line. Wrapping Up In this article, we covered 10 of the most popular frameworks for developing RESTful APIs. We looked at a high-level overview and listed some points for consideration. We also discussed some key factors in how to decide on which API framework to use.
In a Kubernetes context, multi-tenancy refers to sharing one large cluster among multiple teams, applications, or users primarily in lower environments such as development and testing—mainly to reduce cost and operational overhead around managing many clusters. Multi-tenancy is becoming an essential requirement for platform teams deploying Kubernetes infrastructure. Achieving Multi-Tenancy Using Kubernetes Namespaces By far, the most popular approach to achieve multi-tenancy is by using Kubernetes namespaces. Kubernetes namespaces provide an easy way to divide a set of resources, such as pods, services, and deployments, which are only accessible within that namespace. Platform teams usually manage and operate the clusters and have full cluster-level permissions. They accomplish multi-tenancy by creating one or more namespaces specific to each team/application/user and restricting access to those namespaces for end users who are developers, DevOps engineers, and application owners. End users can only perform operations that are specific to the namespaces they have ownership of. This works well for the vast majority of multi-tenancy use cases; however, there is one corner use case when the vcluster framework can be useful. Multi-Tenancy Using the Vcluster Open-Source Framework Restricting end users’ access only to namespaces does not work in some corner cases where the end users need access to cluster-scope objects such as CRDs, ingress controllers, cluster API servers, etc., for their day-to-day development work. Typically, cluster-level access is needed for end users who are involved in developing custom resources and custom controllers to extend Kubernetes API, admission controllers to implement mutating and validating webhooks, and other services that might require custom Kubernetes configuration. Virtual Cluster (Vcluster) is an open-source framework that aims to solve this problem. Vcluster is essentially a virtual cluster that can be created in a physical cluster. Vcluster installs a K3s cluster by default (k0s, k8s, and EKS optionally) in a namespace of the host cluster for every vcluster instance and installs the core Kubernetes components such as API server, controller manager, storage backend, and optionally, a scheduler. End users interact with the virtual cluster API server and get full access to the virtual cluster, still maintaining resource isolation and security as they are restricted to the host namespaces and don’t have access to the host cluster API server. Platform teams create a namespace in the host cluster, configure resource quotas and policies for the host namespace, create a vcluster instance, and hand over the virtual cluster to the end users. Key Questions to Answer Before Deploying Vclusters While deploying vcluster for a small group of end users is fairly simple, platform teams must ask themselves the following questions and implement additional solutions around vcluster to meet their organization’s automation, security, governance, and compliance requirements before making large-scale vcluster deployments: How do you create host-level namespaces and resource quotas for each namespace and map it with internal users/teams? Platform teams still need a solution for namespace-level multi-tenancy, as host namespaces must be created first for deploying vcluster instances. How do you automate the lifecycle management of vcluster for large-scale usage? Platform teams need to solve for things like creation/modification/deletion of vcluster instances, exposing the vcluster API server to end users (using ingress or load balancers), securely distributing vcluster Kubeconfig files to end users, and upgrading vcluster instances (K3s) for software updates and security vulnerabilities. How do you ensure only approved and mandated cluster-wide services are running in each vcluster? Do you deploy them in every vcluster? How do you guarantee there is no drift? These services typically include security plugins, logging, monitoring, service mesh, ingress controllers, storage plugins, etc. How do you create network policies at the namespace level for the host namespaces? This level of network isolation is still needed as the physical cluster may be shared among multiple BUs and application teams—requiring network isolation among host namespaces. How do you enforce security policies such as OPA in each vcluster? Do you deploy these policies in every vcluster? Most of the platform teams standardize a set of security policies based on the recommendations from their security teams and deploy them in every cluster to maintain security posture even in lower environments. How do you retrieve Kubectl audit logs for each vcluster? For some organizations, Kubectl audit is a key requirement, even in lower environments. How do you handle cost allocation? Since the resources are shared by different development teams, they may belong to different cost centers and platform teams need to implement the appropriate chargeback policies for cost allocation. How do you make other developer tools like ArgoCD work with vcluster? GitOps tools like ArgoCD require cluster access for application deployment, so each vcluster instance must be configured in the ArgoCD for end users to leverage ArgoCD GitOps deployment. This may apply to other common tools such as observability, logging, and monitoring tools. What kind of additional resource and operational overhead do you incur with every vcluster instance? Each vcluster essentially is a K3s/K8s cluster with all the add-ons, such as your security plugins, logging plugins, monitoring plugins, ingress controllers, etc. These per vcluster Kubernetes resources and the add-ons can potentially incur substantial overhead as you deploy more vcluster instances. Similarly, since each vcluster essentially is a Kubernetes cluster, platform teams may incur additional overhead managing these clusters for Kubernetes version updates, patch management, and add-on management. Does the vcluster match your real production environment? For some organizations, it’s important that the development environment closely matches the production environment. Vcluster supports other distributions such as K8s and EKS, but platform teams must check whether it’s equivalent to running a standalone cluster for the use cases that require near-production environments. For instance, EKS supports many advanced features, including third-party CNIs, various storage classes, auto-scaling, IRSA, and add-ons, which may not be available in the virtual EKS clusters. Conclusion For most platform teams, multi-tenancy based on namespaces with additional automation around namespace life cycle management, security, and cost control solves their multi-tenancy use cases. Vcluster addresses a specific gap in Kubernetes namespace multi-tenancy whereby end users that don’t have cluster-level privileges can now access cluster-scoped objects within their virtual cluster. Platform teams must internally validate whether there is such a requirement for their end users, do a thorough cost-benefit analysis taking their security, compliance, and governance requirements into consideration, and implement additional automation around it to ensure it’s enterprise-ready before deployment.
We will go over Apache Kafka basics, installation, and operation, as well as a step-by-step implementation using a .NET Core 6 web application. Prerequisites Visual Studio 2022 .NET Core 6 SDK SQL Server Java JDK 11 Apache Kafka Agenda Overview of Event Streaming Introduction to Apache Kafka. Main concepts and foundation of Kafka. Different Kafka APIs. Use cases of Apache Kafka. Installation of Kafka on Windows 10. Step-by-step implementation Overview of Event Streaming Events are the things that happen within our application when we navigate something. For example, we sign up on any website and order something, so, these are the events. The event streaming platform records different types of data like transaction, historical, and real-time data. This platform is also used to process events and allow different consumers to process results immediately and in a timely manner. An event-driven platform allows us to monitor our business and real-time data from different types of devices like IoT and many more. After analyzing, it provides a good customer experience based on different types of events and needs. Introduction to Apache Kafka Below, are a few bullet points that describe Apache Kafka: Kafka is a distributed event store and stream-processing platform. Kafka is open source and is written in Java and Scala. The primary purpose to designed Kafka by Apache foundation is to handle real-time data feeds and provide high throughput and low latency platforms. Kafka is an event streaming platform that has many capabilities to publish (write) and subscribe to (read) streams of events from a different system. Also, to store and process events durably as long as we want, by default, Kafka stores events from seven days of the time period, but we can increase that as per need and requirement. Kafka has distributed system, which has servers and clients that can communicate via TCP protocol. It can be deployed on different virtual machines and containers in on-premise and cloud environments as per requirements. In the Kafka world, a producer sends messages to the Kafka broker. The messages will get stored inside the topics and the consumer subscribes to that topic to consume messages sent by the producer. ZooKeeper is used to manage the metadata of Kafka-related things, it tracks which brokers are part of the Kafka cluster and partitions of different topics. Lastly, it manages the status of Kafka nodes and maintains a list of Kafka topics and messages. Main Concepts and Foundation of Kafka 1. Event An event or record is the message that we read and write to the Kafka server; we do this in the form of events in our business world, and it contains a key, a value, a timestamp, and other metadata headers. The key, value, and time stamp, in this case, are as follows: Key: “Jaydeep” Value: “Booked BMW” Event Timestamp: “Dec. 11, 2022, at 12:00 p.m.” 2. Producer The producer is a client application that sends messages to the Kafka node or broker. 3. Consumer The consumer is an application that receives data from Kafka. 4. Kafka Cluster The Kafka cluster is the set of computers that share the workload with each other with varying purposes. 5. Broker The broker is a Kafka server that acts as an agent between the producer and consumer, who communicate via the broker. 6. Topic The events are stored inside the “topic,” it’s similar to our folder in which we store multiple files. Each topic has one or more producers and consumers, which write and reads data from the topic. Events in “topic” can be read as often as needed because it persists events and it’s not like another messaging system that removes messages after consuming. 7. Partitions Topics are partitions, meaning the topic is spread over multiple partitions that we created inside the topic. When the producer sends some event to the topic, it will store it inside the particular partitions, and then, the consumer can read the event from the corresponding topic partition in sequence. 8. Offset Kafka assigns one unique ID to the message stored inside the topic partition when the message arrives from the producer. 9. Consumer Groups In the Kafka world, the consumer group acts as a single logical unit. 10. Replica In Kafka, to make data fault-tolerant and highly available, we can replicate topics in different regions and brokers. So, in case something wrong happens with data in one topic, we can easily get that from another to replicate the same. Different Kafka APIs Kafka has five core APIs that serve different purposes: Admin API: This API manages different topics, brokers, and Kafka objects. Producer API: This API is used to write/publish events to different Kafka topics. Consumer API: This API is used to receive the different messages corresponding to the topics that are subscribed by the consumer. Kafka Stream API: This API is used to perform different types of operations like windowing, joins, aggregation, and many others. Basically, its use is to transform objects. Kafka Connect API: This API works as a connector to Kafka, which helps different systems connect with Kafka easily. It has different types of ready-to-use connectors related to Kafka. Use Cases of Apache Kafka Messaging User activity tracking Log aggregation Stream processing Realtime data analytics Installation of Kafka on Windows 10 Step 1 Download and install the Java SDK of version 8 or more. Note: I have Java 11, that’s why I put the same path in all commands that I used here. Step 2 Open and install EXE. Step 3 Set the environment variable for Java using the command prompt as admin. Command: setx -m JAVA_HOME “C:\Program Files\Java\jdk-11.0.16.1” setx -m PATH “%JAVA_HOME%\bin;%PATH%” Step 4 After that, download and install Apache Kafka. Step 5 Extract the downloaded Kafka file and rename it “Kafka.” Step 6 Open D:\Kafka\config\ and create a “zookeeper-data” and “kafka-logs” folder inside that. Step 7 Next, open D:\Kafka\config\zookeeper.properties file and add the folder path inside that: D:\Kafka\config\zookeeper.properties dataDir=D:/Kafka/zookeeper-data Step 8 After that, open D:\Kafka\config\server.properties file and change the log path over there: D:\Kafka\config\server.properties log.dirs=D:/Kafka/kafka-logs Step 9 Saves and close both files. Step 10 Run ZooKeeper: D:\Kafka> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties Step 11 Start Kafka: D:\Kafka> .\bin\windows\kafka-server-start.bat .\config\server.properties Step 12 Create Kafka topic: D:\Kafka\bin\windows>kafka-topics.bat — create — bootstrap-server localhost:9092 — replication-factor 1 — partitions 1 — topic testdata Step 13 Create a producer and send some messages after you’ve started a producer and consumer: D:\Kafka\bin\windows>kafka-console-producer.bat — broker-list localhost:9092 — topic testdata Step 14 Next, create a consumer. After, you will see the message the producer sent: D:\Kafka\bin\windows>kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic testdata Step-by-Step Implementation Let’s start with practical implementation. Step 1 Create a new .NET Core Producer Web API: Step 2 Configure your application: Step 3 Provide additional details: Step 4 Install the following two NuGet packages: Step 5 Add configuration details inside the appsettings.json file: JSON { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } }, "AllowedHosts": "*", "producerconfiguration": { "bootstrapservers": "localhost:9092" }, "TopicName": "testdata" } Step 6 Register a few services inside the “Program” class: C# using Confluent.Kafka; var builder = WebApplication.CreateBuilder(args); // Add services to the container. var producerConfiguration = new ProducerConfig(); builder.Configuration.Bind("producerconfiguration", producerConfiguration); builder.Services.AddSingleton<ProducerConfig>(producerConfiguration); builder.Services.AddControllers(); // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.UseSwagger(); app.UseSwaggerUI(); } app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); Step 7 Next, create the CarDetails model class: C# using Microsoft.AspNetCore.Authentication; namespace ProducerApplication.Models { public class CarDetails { public int CarId { get; set; } public string CarName { get; set; } public string BookingStatus { get; set; } } } Step 8 Now, create the CarsController class: C# using Confluent.Kafka; using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Configuration; using Newtonsoft.Json; using ProducerApplication.Models; namespace ProducerApplication.Controllers { [Route("api/[controller]")] [ApiController] public class CarsController : ControllerBase { private ProducerConfig _configuration; private readonly IConfiguration _config; public CarsController(ProducerConfig configuration, IConfiguration config) { _configuration = configuration; _config = config; } [HttpPost("sendBookingDetails")] public async Task<ActionResult> Get([FromBody] CarDetails employee) { string serializedData = JsonConvert.SerializeObject(employee); var topic = _config.GetSection("TopicName").Value; using (var producer = new ProducerBuilder<Null, string>(_configuration).Build()) { await producer.ProduceAsync(topic, new Message<Null, string> { Value = serializedData }); producer.Flush(TimeSpan.FromSeconds(10)); return Ok(true); } } } } Step 9 Finally, run the application and send a message: Step 10 Now, create a “consumer” application: For that, create a new .NET Core console application: Step 11 Configure your application: Step 12 Provide additional information: Step 13 Install the NuGet below: Step 14 Add the following code, which consumes messages sent by the consumer: C# using Confluent.Kafka; var config = new ConsumerConfig { GroupId = "gid-consumers", BootstrapServers = "localhost:9092" }; using (var consumer = new ConsumerBuilder<Null, string>(config).Build()) { consumer.Subscribe("testdata"); while (true) { var bookingDetails = consumer.Consume(); Console.WriteLine(bookingDetails.Message.Value); } } Step 15 Finally, run the producer and consumer, send a message using the producer app, and you will see the message immediately inside the consumer console sent by the producer: Here is the GitHub URL I used in this article. Conclusion Here, we discussed Apache Kafka introduction, working, benefits, and step-by-step implementation using .NET Core 6. Happy Coding!
As much as the JPA Criteria builder is expressive, JPA queries are often equally verbose, and the API itself can be unintuitive to use, especially for newcomers. In the Quarkus ecosystem, Panache is a partial remedy for these problems when using Hibernate. Still, I find myself juggling the Panache’s helper methods, preconfigured enums, and raw strings when composing anything but the simplest of queries. You could claim I am just inexperienced and impatient or, instead, acknowledge that the perfect API is frictionless to use for everyone. Thus, the user experience of writing JPA queries can be further improved in that direction. Introduction One of the remaining shortcomings is that raw strings are inherently not type-safe, meaning my IDE rejects me the helping hand of code completion and wishes me good luck at best. On the upside, Quarkus facilitates application relaunches in a split second to issue quick verdicts on my code. And nothing beats the heart-felt joy and genuine surprise when I have composed a working query on the fifth, rather than the tenth, attempt... With this in mind, we built the open-source library JPAstreamer to make the process of writing Hibernate queries more intuitive and less time-consuming while leaving your existing codebase intact. It achieves this goal by allowing queries to be expressed as standard Java Streams. Upon execution, JPAstreamer translates the stream pipeline to an HQL query for efficient execution and avoids materializing anything but the relevant results. Let me take an example—in some random database exists a table called Person represented in a Hibernate application by the following standard Entity: Java @Entity @Table(name = "person") public class Person { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @Column(name = "person_id", nullable = false, updatable = false) private Integer actorId; @Column(name = "first_name", nullable = false, columnDefinition = "varchar(45)") private String firstName; @Column(name = "last_name", nullable = false, columnDefinition = "varchar(45)") private String lastName; @Column(name = "created_at", nullable = false, updatable = false) private LocalDateTime createdAt; // Getters for all fields will follow from here } To fetch the Person with an Id of 1 using JPAstreamer, all you need is the following: Java @ApplicationScoped public class PersonRepository { @PersistenceContext EntityManagerFactory entityManagerFactory; private final JPAStreamer jpaStreamer; public PersonRepository EntityManagerFactory entityManagerFactory) { jpaStreamer = JPAStreamer.of(entityManagerFactory); <1> } @Override public Optional<Person> getPersonById(int id) { return this.jpaStreamer.from(Person.class) <2> .filter(Person$.personId.equal(id)) <3> .findAny(); } } <1> Initialize JPAstreamer in one line, the underlying JPA provider handles the DB configuration. <2> The stream source is set to be the Person table. <3> The filter operation is treated as an SQL WHERE clause and the condition is expressed type-safely with JPAstreamer predicates (more on this to follow). Despite it looking as if JPAstreamer operates on all Person objects, the pipeline is optimized to a single query, in this case: Plain Text select person0_.person_id as person_id1_0_, person0_.first_name as first_na2_0_, person0_.last_name as last_nam3_0_, person0_.created_at as created_4_0_, from person person0_ where person0_.person_id=1 Thus, only the Person matching the search criteria is ever materialized. Next, we can look at a more complex example in which I am searching for Person’s with a first name ending with an “A” and a last name that starts with “B.” The matches are sorted primarily by first name and secondly by last name. I further decide to apply an offset of 5, excluding the first five results, and to limit the total results to 10. Here is the stream pipeline to achieve this task: Java List<Person> list = jpaStreamer.stream(Person.class) .filter(Person$.firstName.endsWith("A").and(Person$.lastName.startsWith("B"))) <1> .sorted(Person$.firstName.comparator().thenComparing(Person$.lastName.comparator())) <2> .skip(5) <3> .limit(10) <4> .collect(Collectors.toList()) <1> Filters can be combined with the and/or operators. <2> Easily filter on one or more properties. <3> Skip the first 5 Persons. <4> Return at most 10 Persons. In the context of queries, the stream operators filter, sort, limit, and skip, all have a natural mapping that makes the resulting query expressive and intuitive to read while remaining compact. This query is translated by JPAstreamer to the following HQL statement: Plain Text select person0_.person_id as person_id1_0_, person0_.first_name as first_na2_0_, person0_.last_name as last_nam3_0_, person0_.created_at as created_4_0_, from person person0_ where person0_.person_id=1 where (person0_.first_name like ?) and (person0_.last_name like ?) order by person0_.first_name asc, person0_.last_name asc limit ?, ? How JPAstreamer Works Okay, it looks simple. But how does it work? JPAstreamer uses an annotation processor to form a meta-model at compile time. It inspects any classes marked with the standard JPA annotation @Entity, and for every entity Foo.class, a corresponding Foo$.class is created. The generated classes represent entity attributes as Fields used to form predicates on the form User$.firstName.startsWith("A") that can be interpreted by JPAstreamer’s query optimizer. It is worth repeating that JPAstreamer does not alter or disturb the existing codebase but merely extends the API to handle Java stream queries. Installing the JPAstreamer Extension JPAstreamer is installed as any other Quarkus extension, using a Maven dependency: XML <dependency> <groupId>io.quarkiverse.jpastreamer</groupId> <artifactId>quarkus-jpastreamer</artifactId> <version>1.0.0</version> </dependency> After the dependency is added, rebuild your Quarkus application to trigger JPAstreamer’s annotation processor. The installation is complete once the generated fields reside in /target/generated-sources; you’ll recognize them by the trailing $ in the class names, e.g., Person$.class. Note: JPAstreamer requires an underlying JPA provider, such as Hibernate. For this reason, JPAstreamer needs no additional configuration as the database integration is taken care of by the JPA provider. JPAstreamer and Panache Any Panache fan will note that JPAstreamer shares some of its objectives with Panache, in simplifying many common queries. Still, JPAstreamer distinguishes itself by instilling more confidence in the queries with its type-safe stream interface. However, no one is forced to take a pick as Panache and JPAstreamer work seamlessly alongside each other. Note: Here is an example Quarkus application that uses both JPAstreamer and Panache. At the time of writing, JPAstreamer does not have support for Panache’s Active Record Pattern, as it relies on standard JPA Entities to generate its meta-model. This will likely change in the near future. Summary JPA in general, and Hibernate have greatly simplified application database access, but its API sometimes forces unnecessary complexity. With JPAstreamer, you can utilize JPA while keeping your codebase clean and maintainable.
Trino is an open-source distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources. Trino was designed to handle data warehousing, ETL, and interactive analytics by large amounts of data and producing reports. Alluxio is an open-source data orchestration platform for large-scale analytics and AI. Alluxio sits between compute frameworks such as Trino and Apache Spark and various storage systems like Amazon S3, Google Cloud Storage, HDFS, and MinIO. This is a tutorial for deploying Alluxio as the caching layer for Trino using the Iceberg connector. Why Do We Need Caching for Trino? A small fraction of the petabytes of data you store is generating business value at any given time. Repeatedly scanning the same data and transferring it over the network consumes time, compute cycles, and resources. This issue is compounded when pulling data from disparate Trino clusters across regions or clouds. In these circumstances, caching solutions can significantly reduce the latency and cost of your queries. Trino has a built-in caching engine, Rubix, in its Hive connector. While this system is convenient as it comes with Trino, it is limited to the Hive connector and has not been maintained since 2020. It also lacks security features and support for additional compute engines. Trino on Alluxio Alluxio connects Trino to various storage systems, providing APIs and a unified namespace for data-driven applications. Alluxio allows Trino to access data regardless of the data source and transparently cache frequently accessed data (e.g., tables commonly used) into Alluxio distributed storage. Using Alluxio Caching via the Iceberg Connector Over MinIO File Storage We’ve created a demo that demonstrates how to configure Alluxio to use write-through caching with MinIO. This is achieved by using the Iceberg connector and making a single change to the location property on the table from the Trino perspective. In this demo, Alluxio is run on separate servers; however, it’s recommended to run it on the same nodes as Trino. This means that all the configurations for Alluxio will be located on the servers where Alluxio runs, while Trino’s configuration remains unaffected. The advantage of running Alluxio externally is that it won’t compete for resources with Trino, but the disadvantage is that data will need to be transferred over the network when reading from Alluxio. It is crucial for performance that Trino and Alluxio are on the same network. To follow this demo, copy the code located here. Trino Configuration Trino is configured identically to a standard Iceberg configuration. Since Alluxio is running external to Trino, the only configuration needed is at query time and not at startup. Alluxio Configuration The configuration for Alluxio can all be set using the alluxio-site.properties file. To keep all configurations colocated on the docker-compose.yml, we are setting them using Java properties via the ALLUXIO_JAVA_OPTS environment variable. This tutorial also refers to the master node as the leader and the workers as followers. Master Configurations alluxio.master.mount.table.root.ufs=s3://alluxio/ The leader exposes ports 19998 and 19999, the latter being the port for the web UI. Worker Configurations alluxio.worker.ramdisk.size=1G alluxio.worker.hostname=alluxio-follower The follower exposes ports 29999 and 30000, and sets up a shared memory used by Alluxio to store data. This is set to 1G via the shm_size property and is referenced from the alluxio.worker.ramdisk.size property. Shared Configurations Between Leader and Follower alluxio.master.hostname=alluxio-leader # Minio configs alluxio.underfs.s3.endpoint=http://minio:9000 alluxio.underfs.s3.disable.dns.buckets=true alluxio.underfs.s3.inherit.acl=false aws.accessKeyId=minio aws.secretKey=minio123 # Demo-only configs alluxio.security.authorization.permission.enabled=false The alluxio.master.hostname needs to be on all nodes, leaders and followers. The majority of shared configs points Alluxio to the underfs, which is MinIO in this case. alluxio.security.authorization.permission.enabled is set to “false” to keep the Docker setup simple. Note: This is not recommended to do in a production or CI/CD environment. Running Services First, you want to start the services. Make sure you are in the trino-getting-started/iceberg/trino-alluxio-iceberg-minio directory. Now, run the following command: docker-compose up -d You should expect to see the following output. Docker may also have to download the Docker images before you see the “Created/Started” messages, so there could be extra output: [+] Running 10/10 ⠿ Network trino-alluxio-iceberg-minio_trino-network Created 0.0s ⠿ Volume "trino-alluxio-iceberg-minio_minio-data" Created 0.0s ⠿ Container trino-alluxio-iceberg-minio-mariadb-1 Started 0.6s ⠿ Container trino-alluxio-iceberg-minio-trino-coordinator-1 Started 0.7s ⠿ Container trino-alluxio-iceberg-minio-alluxio-leader-1 Started 0.9s ⠿ Container minio Started 0.8s ⠿ Container trino-alluxio-iceberg-minio-alluxio-follower-1 Started 1.5s ⠿ Container mc Started 1.4s ⠿ Container trino-alluxio-iceberg-minio-hive-metastore-1 Started Open Trino CLI Once this is complete, you can log into the Trino coordinator node. We will do this by using the exec command and run the trino CLI executable as the command we run on that container. Notice the container id is trino-alluxio-iceberg-minio-trino-coordinator-1, so the command you will run is: <<<<<<< HEAD docker container exec -it trino-alluxio-iceberg-minio-trino-coordinator-1 trino ======= docker container exec -it trino-minio_trino-coordinator_1 trino >>>>>>> alluxio When you start this step, you should see the trino cursor once the startup is complete. It should look like this when it is done: trino> To best understand how this configuration works, let’s create an Iceberg table using a CTAS (CREATE TABLE AS) query that pushes data from one of the TPC connectors into Iceberg that points to MinIO. The TPC connectors generate data on the fly so we can run simple tests like this. First, run a command to show the catalogs to see the tpch and iceberg catalogs since these are what we will use in the CTAS query: SHOW CATALOGS; You should see that the Iceberg catalog is registered. MinIO Buckets and Trino Schemas Upon startup, the following command is executed on an intiailization container that includes the mc CLI for MinIO. This creates a bucket in MinIO called /alluxio, which gives us a location to write our data to and we can tell Trino where to find it: /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 minio minio123) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/alluxio; /usr/bin/mc mb minio/alluxio; /usr/bin/mc policy set public minio/alluxio; exit 0; " Note: This bucket will act as the mount point for Alluxio, so the schema directory alluxio://lakehouse/ in Alluxio will map to s3://alluxio/lakehouse/. Querying Trino Let’s move to creating our SCHEMA that points us to the bucket in MinIO and then run our CTAS query. Back in the terminal, create the iceberg.lakehouse SCHEMA. This will be the first call to the metastore to save the location of the schema location in the Alluxio namespace. Notice, we will need to specify the hostname alluxio-leader and port 19998 since we did not set Alluxio as the default file system. Take this into consideration if you want Alluxio caching to be the default usage and transparent to users managing DDL statements: CREATE SCHEMA iceberg.lakehouse WITH (location = 'alluxio://alluxio-leader:19998/lakehouse/'); Now that we have a SCHEMA that references the bucket where we store our tables in Alluxio, which syncs to MinIO, we can create our first table. Optional: To view your queries run, log into the Trino UI and log in using any username (it doesn’t matter since no security is set up). Move the customer data from the tiny generated TPCH data into MinIO using a CTAS query. Run the following query, and if you like, watch it running on the Trino UI: CREATE TABLE iceberg.lakehouse.customer WITH ( format = 'ORC', location = 'alluxio://alluxio-leader:19998/lakehouse/customer/' ) AS SELECT * FROM tpch.tiny.customer; Go to the Alluxio UI and the MinIO UI, and browse the Alluxio and MinIO files. You will now see a lakehouse directory that contains a customer directory that contains the data written by Trino to Alluxio and Alluxio writing it to MinIO. Now, there is a table under Alluxio and MinIO, you can query this data by checking the following: SELECT * FROM iceberg.lakehouse.customer LIMIT 10; How are we sure that Trino is actually reading from Alluxio and not MinIO? Let’s delete the data in MinIO and run the query again just to be sure. Once you delete this data, you should still see data return. Stopping Services Once you complete this tutorial, the resources used for this excercise can be released by runnning the following command: docker-compose down Conclusion At this point, you should have a better understanding of Trino and Alluxio, how to get started with deploying Trino and Alluxio, and how to use Alluxio caching with an Iceberg connector and MinIO file storage. I hope you enjoyed this article. Be sure to like this article and comment if you have any questions!
In earlier days, it was easy to deduct and debug a problem in monolithic applications because there was only one service running in the back end and front end. Now, we are moving toward microservices architecture, where applications are divided into multiple independently deployable services. These services have their own goal and logic to serve. In this kind of application architecture, it becomes difficult to observe how one service depends on or affects other services. To make the system observable, some logs, metrics, or traces must be emitted from the code, and this data must be sent to an observability back end. This is where OpenTelemetry and Jaeger come into the picture. In this article, we will see how to monitor application trace data (Traces and Spans) with the help of OpenTelemetry and Jaeger. A trace is used to observe the requests as they propagate through the services in a distributed system. Spans are a basic unit of the trace; they represent a single event within the trace, and a trace can have one or multiple spans. A span consists of log messages, time-related data, and other attributes to provide information about the operation it tracks. We will use the distributed tracing method to observe requests moving across microservices, generating data about the request and making it available for analysis. The produced data will have a record of the flow of requests in our microservices, and it will help us understand our application's performance. OpenTelemetry Telemetry is the collection and transmission of data using agents and protocols from the source in observability. The telemetry data includes logs, metrics, and traces, which help us understand what is happening in our application. OpenTelemetry (also known as OTel) is an open source framework comprising a collection of tools, APIs, and SDKs. OpenTelemetry makes generating, instrumenting, collecting, and exporting telemetry data easy. The data collected from OpenTelemetry is vendor-agnostic and can be exported in many formats. OpenTelemetry is formed after merging two projects OpenCensus and OpenTracing. Instrumenting The process of adding observability code to your application is known as instrumentation. Instrumentation helps make our application observable, meaning the code must produce some metrics, traces, and logs. OpenTelemetry provides two ways to instrument our code: Manual instrumentation Auto instrumentation 1. Manual Instrumentation The user needs to add an OpenTelemetry code to the application. The manual instrumentation provides more options for customization in spans and traces. Languages supported for manual instrumentations are C++, .NET, Go, Java, Python, and so on. 2. Automatic Instrumentation This is the easiest way of instrumentation as it requires no code changes and no need to recompile the application. It uses an intelligent agent that gets attached to an application, reads its activity, and extracts the traces. Automatic instrumentation supports Java, NodeJS, Python, and so on. Difference Between Manual and Automatic Instrumentation Both manual and automatic instrumentation have advantages and disadvantages that you might consider while writing your code. A few of them are listed below: Manual Instrumentation Automatic Instrumentation Code changes are required. Code changes are not required. It supports maximum programming languages. Currently, .Net, Java, NodeJS, and Python are supported. It consumes a lot of time as code changes are required. Easy to implement as we do not need to touch the code. Provide more options for the customization of spans and traces. As you have more control over the telemetry data generated by your application. Fewer options for customization. Possibilities of error are high as manual changes are required. No error possibilities. As we don't have to touch our application code. To make the instrumentation process hassle-free, use automatic instrumentation, as it does not require any modification in the code and reduces the possibility of errors. Automatic instrumentation is done by an agent which reads your application's telemetry data, so no manual changes are required. For the scope of this post, we will see how you can use automatic instrumentation in a Kubernetes-based microservices environment. Jaeger Jaeger is a distributed tracing tool initially built by Uber and released as open-source in 2015. Jaeger is also a Cloud Native Computing Foundation graduate project and was influenced by Dapper and OpenZipkin. It is used for monitoring and troubleshooting microservices-based distributed systems. The Jaeger components which we have used for this blog are: Jaeger Collector Jaeger Query Jaeger UI / Console Storage Backend Jaeger Collector: The Jaeger distributed tracing system includes the Jaeger collector. It is in charge of gathering and keeping the information. After receiving spans, the collector adds them to a processing queue. Collectors need a persistent storage backend, hence Jaeger also provides a pluggable span storage mechanism. Jaeger Query: This is a service used to get traces out of storage. The web-based user interface for the Jaeger distributed tracing system is called Jaeger Query. It provides various features and tools to help you understand the performance and behavior of your distributed application and enables you to search, filter, and visualise the data gathered by Jaeger. Jaeger UI/Console: Jaeger UI lets you view and analyze traces generated by your application. Storage Back End: This is used to store the traces generated by an application for the long term. In this article, we are going to use Elasticsearch to store the traces. What Is the Need for Integrating OpenTelemetry With Jaeger? OpenTelemetry and Jaeger are the tools that help us in setting the observability in microservices-based distributed systems, but they are intended to address different issues. OpenTelemetry provides an instrumentation layer for the application, which helps us generate, collect and export the telemetry data for analysis. In contrast, Jaeger is used to store and visualize telemetry data. OpenTelemetry can only generate and collect the data. It does not have a UI for the visualization. So we need to integrate Jaeger with OpenTelemetry as it has a storage backend and a web UI for the visualization of the telemetry data. With the help of Jaeger UI, we can quickly troubleshoot microservices-based distributed systems. Note: OpenTelemetry can generate logs, metrics, and traces. Jaeger does not support logs and metrics. Now you have an idea about OpenTelemetry and Jaeger. Let's see how we can integrate them with each other to visualize the traces and spans generated by our application. Implementing OpenTelemetry Auto-Instrumentation We will integrate OpenTelemetry with Jaeger, where OpenTelemetry will act as an instrumentation layer for our application, and Jaeger will act as the back-end analysis tool to visualize the trace data. Jaeger will get the telemetry data from the OpenTelemetry agent. It will store the data in the storage backend, from where we will query the stored data and visualize it in the Jaeger UI. Prerequisites for this article are: The target Kubernetes cluster is up and running. You have access to run the kubectl command against the Kubernetes cluster to deploy resources. Cert manager is installed and running. You can install it from the website cert-manager.io if it is not installed. We assume that you have all the prerequisites and now you are ready for the installation. The files we have used for this post are available in this GitHub repo. Installation The installation part contains three steps: Elasticsearch installation Jaeger installation OpenTelemetry installation Elasticsearch By default, Jaeger uses in-memory storage to store spans, which is not a recommended approach for the production environment. There are various tools available to use as a storage back end in Jaeger; you can read about them in the official documentation of Jaeger span storage back end. In this article, we will use Elasticsearch as a storage back end. You can deploy Elasticsearch in your Kubernetes cluster using the Elasticsearch Helm chart. While deploying Elasticsearch, ensure you have enabled the password-based authentication and deploy that Elasticsearch in observability namespaces. Elasticsearch is deployed in our Kubernetes cluster, and you can see the output by running the following command. Shell $ kubectl get all -n observability NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 17m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m NAME READY AGE statefulset.apps/elasticsearch 1/1 17m Jaeger Installation We are going to use Jaeger to visualize the trace data. Let's deploy the Jaeger Operator on our cluster. Before proceeding with the installation, we will deploy a ConfigMap in the observability namespace. In this ConfigMap, we will pass the username and password of the Elasticsearch which we have deployed in the previous step. Replace the credentials based on your setup. YAML kubectl -n observability apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: jaeger-configuration labels: app: jaeger app.kubernetes.io/name: jaeger data: span-storage-type: elasticsearch collector: | es: server-urls: http://elasticsearch:9200 username: elastic password: changeme collector: zipkin: http-port: 9411 query: | es: server-urls: http://elasticsearch:9200 username: elastic password: changeme agent: | collector: host-port: "jaeger-collector:14267" EOF If you are going to deploy Jaeger in another namespace and you have changed the Jaeger collector service name, then you need to change the values of the host-port value under the agent collector. Jaeger Operator The Jaeger Operator is a Kubernetes operator for deploying and managing Jaeger, an open source, distributed tracing system. It works by automating the deployment, scaling, and management of Jaeger components on a Kubernetes cluster. The Jaeger Operator uses custom resources and custom controllers to extend the Kubernetes API with Jaeger-specific functionality. It manages the creation, update, and deletion of Jaeger components, such as the Jaeger collector, query, and agent components. When a Jaeger instance is created, the Jaeger Operator deploys the necessary components and sets up the required services and configurations. We are going to deploy the Jaeger Operator in the observability namespace. Use the below-mentioned command to deploy the operator. Shell $ kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.38.0/jaeger-operator.yaml -n observability We are using the latest version of Jaeger, which is 1.38.0 at the time of writing this article. By default, the Jaeger script is provided for cluster-wide mode. Suppose you want to watch only a particular namespace. In that case, you need to change the ClusterRole to Role and ClusterBindingRole to RoleBinding in the operator manifest and set the WATCH_NAMESPACE env variable on the Jaeger Operator deployment. To verify whether Jaeger is deployed successfully or not, run the following command: Shell $ kubectl get all -n observability NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 17m pod/jaeger-operator-5597f99c79-hd9pw 2/2 Running 0 11m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m service/jaeger-operator-metrics ClusterIP 172.20.220.212 <none> 8443/TCP 11m service/jaeger-operator-webhook-service ClusterIP 172.20.224.23 <none> 443/TCP 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/jaeger-operator 1/1 1 1 11m NAME DESIRED CURRENT READY AGE replicaset.apps/jaeger-operator-5597f99c79 1 1 1 11m NAME READY AGE statefulset.apps/elasticsearch 1/1 17m As we can see in the above output, our Jaeger Operator is deployed successfully, and all of its pods are up and running; this means Jaeger Operator is ready to install the Jaeger instances (CRs). The Jaeger instance will contain Jaeger components (Query, Collector, Agent); later, we will use these components to query OpenTelemetry metrics. Jaeger Instance A Jaeger Instance is a deployment of the Jaeger distributed tracing system. It is used to collect and store trace data from microservices or distributed applications, and provide a UI to visualize and analyze the trace data. To deploy the Jaeger instance, use the following command. Shell $ kubectl apply -f https://raw.githubusercontent.com/infracloudio/Opentelemertrywithjaeger/master/jaeger-production-template.yaml To verify the status of the Jaeger instance, run the following command: Shell $ kubectl get all -n observability NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 17m pod/jaeger-agent-27fcp 1/1 Running 0 14s pod/jaeger-agent-6lvp2 1/1 Running 0 15s pod/jaeger-collector-69d7cd5df9-t6nz9 1/1 Running 0 19s pod/jaeger-operator-5597f99c79-hd9pw 2/2 Running 0 11m pod/jaeger-query-6c975459b6-8xlwc 1/1 Running 0 16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m service/jaeger-collector ClusterIP 172.20.24.132 <none> 14267/TCP,14268/TCP,9411/TCP,14250/TCP 19s service/jaeger-operator-metrics ClusterIP 172.20.220.212 <none> 8443/TCP 11m service/jaeger-operator-webhook-service ClusterIP 172.20.224.23 <none> 443/TCP 11m service/jaeger-query LoadBalancer 172.20.74.114 a567a8de8fd5149409c7edeb54bd39ef-365075103.us-west-2.elb.amazonaws.com 80:32406/TCP 16s service/zipkin ClusterIP 172.20.61.72 <none> 9411/TCP 18s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/jaeger-agent 2 2 2 2 2 <none> 16s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/jaeger-collector 1/1 1 1 21s deployment.apps/jaeger-operator 1/1 1 1 11m deployment.apps/jaeger-query 1/1 1 1 18s NAME DESIRED CURRENT READY AGE replicaset.apps/jaeger-collector-69d7cd5df9 1 1 1 21s replicaset.apps/jaeger-operator-5597f99c79 1 1 1 11m replicaset.apps/jaeger-query-6c975459b6 1 1 1 18s NAME READY AGE statefulset.apps/elasticsearch 1/1 17m As we can see in the above screenshot, our Jaeger instance is up and running. OpenTelemetry To install the OpenTelemetry, we need to install the OpenTelemetry Operator. The OpenTelemetry Operator uses custom resources and custom controllers to extend the Kubernetes API with OpenTelemetry-specific functionality, making it easier to deploy and manage the OpenTelemetry observability stack in a Kubernetes environment. The operator manages two things: Collectors: It offers a vendor-agnostic implementation of how to receive, process, and export telemetry data. Auto-instrumentation of the workload using OpenTelemetry instrumentation libraries. It does not require the end-user to modify the application source code. OpenTelemetry Operator To implement the auto-instrumentation, we need to deploy the OpenTelemetry operator on our Kubernetes cluster. To deploy the k8s operator for OpenTelemetry, follow the K8s operator documentation. You can verify the deployment of the OpenTelemetry operator by running the below-mentioned command: Shell $ kubectl get all -n opentelemetry-operator-system NAME READY STATUS RESTARTS AGE pod/opentelemetry-operator-controller-manager-7f479c786d-zzfd8 2/2 Running 0 30s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/opentelemetry-operator-controller-manager-metrics-service ClusterIP 172.20.70.244 <none> 8443/TCP 32s service/opentelemetry-operator-webhook-service ClusterIP 172.20.150.120 <none> 443/TCP 31s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/opentelemetry-operator-controller-manager 1/1 1 1 31s NAME DESIRED CURRENT READY AGE replicaset.apps/opentelemetry-operator-controller-manager-7f479c786d 1 1 1 31s As we can see in the above output, the opentelemetry-operator-controller-manager deployment is running in the opentelemetry-operator-system namespace. OpenTelemetry Collector The OpenTelemetry facilitates the collection of telemetry data via the OpenTelemetry Collector. Collector offers a vendor-agnostic implementation on how to receive, process, and export the telemetry data. The collector is made up of the following components: Receivers: It manages how to get data into the collector. Processors: It manages the processing of data. Exporters: Responsible for sending the received data. We also need to export the telemetry data to the Jaeger instance. Use the following manifest to deploy the collector. YAML kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: config: | receivers: otlp: protocols: grpc: http: processors: exporters: logging: jaeger: endpoint: "jaeger-collector.observability.svc.cluster.local:14250" tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [] exporters: [logging, jaeger] EOF In the above code, the Jaeger endpoint is the address of the Jaeger service which is running inside the observability namespace. We need to deploy this manifest in the same namespace where our application is deployed, so that it can fetch the traces from the application and export them to Jaeger. To verify the deployment of the collector, run the following command. Shell $ kubectl get deploy otel-collector NAME READY UP-TO-DATE AVAILABLE AGE otel-collector 1/1 1 1 41s OpenTelemetry Auto-Instrumentation Injection The above-deployed operator can inject and configure the auto-instrumentation libraries of OpenTelemetry into an application's codebase as it runs. To enable the auto-instrumentation on our cluster, we need to configure an instrumentation resource with the configuration for the SDK and instrumentation. Use the below-given manifest to create the auto-instrumentation. YAML kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: exporter: endpoint: http://otel-collector:4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "0.25" EOF In the above manifest, we have used three things: exporter, propagator, and sampler. Exporter: Used to send data to OpenTelemetry collector at the specified endpoint. In our scenario, it is "http://otel-collector:4317". Propagators: Carry traces, context, and baggage data between distributed tracing systems. They have three propagation mechanisms: tracecontext: This refers to the W3C Trace Context specification, which defines a standard way to propagate trace context information between services. baggage: This refers to the OpenTelemetry baggage mechanism, which allows for the propagation of arbitrary key-value pairs along with the trace context information. b3: This refers to the B3 header format, which is a popular trace context propagation format used by the Zipkin tracing system. Sampler: Uses a "parent-based trace ID ratio" strategy with a sample rate of 0.25 (25%). This means that when tracing a request, if any of its parent requests has already been sampled (with a probability of 0.25), then this request will also be sampled, otherwise it will not be traced. To verify that our custom resource is created or not, we can use the below-mentioned command. Shell $ kubectl get otelinst NAME AGE ENDPOINT SAMPLER SAMPLER ARG my-instrumentation 6s http://otel-collector:4317 parentbased_traceidratio 0.25 This means our custom resource is created successfully. We are using the OpenTelemetry auto-instrumented method, so we don’t need to write instrumentation code in our application. All we need to do is, add an annotation in the pod of our application for auto-instrumentation. As we are going to demo a Java application, the annotation that we will have to use here is: Shell instrumentation.opentelemetry.io/inject-java: "true" Note: The annotation can be added to a namespace as well so that all pods within that namespace will get instrumentation, or by adding the annotation to individual PodSpec objects, available as part of Deployment, Statefulset, and other resources. Below is an example of how your manifest will look after adding the annotations. In the below example, we are using annotation for a Java application. YAML apiVersion: apps/v1 kind: Deployment metadata: name: demo-sagar spec: replicas: 1 selector: matchLabels: app: demo-sagar template: metadata: labels: app: demo-sagar annotations: instrumentation.opentelemetry.io/inject-java: "true" instrumentation.opentelemetry.io/container-names: "spring" spec: containers: - name: spring image: sagar27/petclinic-demo ports: - containerPort: 8080 We have added instrumentation “inject-java” and “container-name” under annotations. If you have multiple container pods, you can add them in the same “container-names” annotation, separated by a comma. For example, “container-name1,container-name-2,container-name-3” etc. After adding the annotations, deploy your application and access it on the browser. Here in our scenario, we are using port-forward to access the application. Shell $ kubectl port-forward service/demo-sagar 8080:8080 To generate traces, either you can navigate through all the pages of this website or you can use the following Bash script: Shell while true; do curl http://localhost:8080/ curl http://localhost:8080/owners/find curl http://localhost:8080/owners?lastName= curl http://localhost:8080/vets.html curl http://localhost:8080/oups curl http://localhost:8080/oups sleep 0.01 done The above-given script will make a curl request to all the pages of the website, and we will see the traces of the request on the Jaeger UI. We are making curl requests to https://localhost:8080 because we use the port-forwarding technique to access the application. You can make changes in the Bash script according to your scenario. Now let’s access the Jaeger UI, as our service jaeger-query uses service type LoadBalancer, we can access the Jaeger UI on the browser by using the load balancer domain/IP. Paste the load balancer domain/IP on the browser and you will see the Jaeger UI there. We have to select our app from the service list and it will show us the traces it generates. In the above screenshot, we have selected our app name “demo-sagar” under the services option and its traces are visible on Jaeger UI. We can further click on the traces to get more details about it. Summary In this article, we have gone through how you can easily instrument your application using the OpenTelemetry auto-instrumentation method. We also learned how this telemetric data could be exported to the Elasticsearch backend and visualized it using Jaeger. Integrating OpenTelemetry with Jaeger will help you in monitoring and troubleshooting. It also helps perform root cause analysis of any bug/issues in your microservice-based distributed systems, performance/latency optimization, service dependency analysis, and so on. We hope you found this post informative and engaging. References OpenTelemetry Jaeger Tracing
GitOps is a software development and operations methodology that uses Git as the source of truth for deployment configurations. It involves keeping the desired state of an application or infrastructure in a Git repository and using Git-based workflows to manage and deploy changes. Two popular open-source tools that help organizations implement GitOps for managing their Kubernetes applications are Flux and Argo CD. In this article, we’ll take a closer look at these tools, their pros and cons, and how to set them up. Common Use Cases for Flux and Argo CD Flux Continuous delivery: Flux can be used to automate the deployment pipeline and ensure that changes are automatically deployed as soon as they are pushed to the Git repository. Configuration management: Flux allows you to store and manage your application’s configuration as code, making it easier to version control and track changes. Immutable infrastructure: Flux helps enforce an immutable infrastructure approach—where changes are made only through the Git repository and not through manual intervention on the cluster. Blue-green deployments: Flux supports blue-green deployments—where a new version of an application is deployed alongside the existing version, and traffic is gradually shifted to the new version. Argo CD Continuous deployment: Argo CD can be used to automate the deployment process, ensuring that applications are always up-to-date with the latest changes from the Git repository. Application promotion: Argo CD supports application promotion—where applications can be promoted from one environment to another. For example, from development to production. Multi-cluster management: Argo CD can be used to manage applications across multiple clusters, ensuring the desired state of the applications is consistent across all clusters. Rollback management: Argo CD provides rollback capabilities, making it easier to revert changes in case of failures. The choice between the two tools depends on the specific requirements of the organization and application, but both tools provide a GitOps approach to simplify the deployment process and reduce the risk of manual errors. They both have their own pros and cons, and in this article, we’ll take a look at what they are and how to set them up. What Is Flux? Flux is a GitOps tool that automates the deployment of applications on Kubernetes. It works by continuously monitoring the state of a Git repository and applying any changes to a cluster. Flux integrates with various Git providers such as GitHub, GitLab, and Bitbucket. When changes are made to the repository, Flux automatically detects them and updates the cluster accordingly. Pros of Flux Automated deployments: Flux automates the deployment process, reducing manual errors and freeing up developers to focus on other tasks. Git-based workflow: Flux leverages Git as a source of truth, which makes it easier to track and revert changes. Declarative configuration: Flux uses Kubernetes manifests to define the desired state of a cluster, making it easier to manage and track changes. Cons of Flux Limited customization: Flux only supports a limited set of customizations, which may not be suitable for all use cases. Steep learning curve: Flux has a steep learning curve for new users and requires a deep understanding of Kubernetes and Git. How To Set Up Flux Prerequisites A running Kubernetes cluster. Helm installed on your local machine. A Git repository for your application's source code and Kubernetes manifests. The repository URL and a SSH key for the Git repository. Step 1: Add the Flux Helm Repository The first step is to add the Flux Helm repository to your local machine. Run the following command to add the repository: Shell helm repo add fluxcd https://charts.fluxcd.io Step 2: Install Flux Now that the Flux Helm repository is added, you can install Flux on the cluster. Run the following command to install Flux: Shell helm upgrade -i flux fluxcd/flux \ --set git.url=git@github.com:<your-org>/<your-repo>.git \ --set git.path=<path-to-manifests> \ --set git.pollInterval=1m \ --set git.ssh.secretName=flux-git-ssh In the above command, replace the placeholder values with your own Git repository information. The git.url parameter is the URL of the Git repository, the git.path parameter is the path to the directory containing the Kubernetes manifests, and the git.ssh.secretName parameter is the name of the SSH secret containing the SSH key for the repository. Step 3: Verify the Installation After running the above command, you can verify the installation by checking the status of the Flux pods. Run the following command to view the pods: Shell kubectl get pods -n <flux-namespace> If the pods are running, Flux has been installed successfully. Step 4: Connect Flux to Your Git Repository The final step is to connect Flux to your Git repository. Run the following command to generate a SSH key and create a secret: Shell ssh-keygen -t rsa -b 4096 -f id_rsa kubectl create secret generic flux-git-ssh \ --from-file=id_rsa=./id_rsa --namespace=<flux-namespace> In the above command, replace the <flux-namespace> placeholder with the namespace where Flux is installed. Now, add the generated public key as a deployment key in your Git repository. You have successfully set up Flux using Helm. Whenever changes are made to the Git repository, Flux will detect them and update the cluster accordingly. In conclusion, setting up Flux using Helm is a quite simple process. By using Git as a source of truth and continuously monitoring the state of the cluster, Flux helps simplify the deployment process and reduce the risk of manual errors. What Is Argo CD? Argo CD is an open-source GitOps tool that automates the deployment of applications on Kubernetes. It allows developers to declaratively manage their applications and keeps the desired state of the applications in sync with the live state. Argo CD integrates with Git repositories and continuously monitors them for changes. Whenever changes are detected, Argo CD applies them to the cluster, ensuring the application is always up-to-date. With Argo CD, organizations can automate their deployment process, reduce the risk of manual errors, and benefit from Git’s version control capabilities. Argo CD provides a graphical user interface and a command-line interface, making it easy to use and manage applications at scale. Pros of Argo CD Advanced deployment features: Argo CD provides advanced deployment features, such as rolling updates and canary deployments, making it easier to manage complex deployments. User-friendly interface: Argo CD provides a user-friendly interface that makes it easier to manage deployments, especially for non-technical users. Customizable: Argo CD allows for greater customization, making it easier to fit the tool to specific use cases. Cons of Argo CD Steep learning curve: Argo CD has a steep learning curve for new users and requires a deep understanding of Kubernetes and Git. Complexity: Argo CD has a more complex architecture than Flux, which can make it more difficult to manage and troubleshoot. How To Set Up Argo CD Argo CD can be installed on a Kubernetes cluster using Helm, a package manager for Kubernetes. In this section, we’ll go through the steps to set up Argo CD using Helm. Prerequisites A running Kubernetes cluster. Helm installed on your local machine. A Git repository for your application’s source code and Kubernetes manifests. Step 1: Add the Argo CD Helm Repository The first step is to add the Argo CD Helm repository to your local machine. Run the following command to add the repository: Shell helm repo add argo https://argoproj.github.io/argo-cd Step 2: Install Argo CD Now that the Argo CD Helm repository is added, you can install Argo CD on the cluster. Run the following command to install Argo CD: Shell helm upgrade -i argocd argo/argo-cd --set server.route.enabled=true Step 3: Verify the Installation After running the above command, you can verify the installation by checking the status of the Argo CD pods. Run the following command to view the pods: Shell kubectl get pods -n argocd If the pods are running, Argo CD has been installed successfully. Step 4: Connect Argo CD to Your Git Repository The final step is to connect Argo CD to your Git repository. Argo CD provides a graphical user interface that you can use to create applications and connect to your Git repository. To access the Argo CD interface, run the following command to get the URL: Shell kubectl get routes -n argocd Use the URL in a web browser to access the Argo CD interface. Once you’re in the interface, you can create a new application by providing the Git repository URL and the path to the Kubernetes manifests. Argo CD will continuously monitor the repository for changes and apply them to the cluster. You have now successfully set up Argo CD using Helm. Conclusion GitOps is a valuable approach for automating the deployment and management of applications on Kubernetes. Flux and Argo CD are two popular GitOps tools that provide a simple and efficient way to automate the deployment process, enforce an immutable infrastructure, and manage applications in a consistent and predictable way. Flux focuses on automating the deployment pipeline and providing configuration management as code, while Argo CD provides a more complete GitOps solution, including features such as multi-cluster management, application promotion, and rollback management. Both tools have their own strengths and weaknesses, and the choice between the two will depend on the specific requirements of the organization and the application. Regardless of the tool chosen, GitOps provides a valuable approach for simplifying the deployment process and reducing the risk of manual errors. By keeping the desired state of the applications in sync with the Git repository, GitOps ensures that changes are made in a consistent and predictable way, resulting in a more reliable and efficient deployment process.
Mark Gardner
Independent Contractor,
The Perl Shop
Nuwan Dias
VP and Deputy CTO,
WSO2
Radivoje Ostojic
Principal Software Engineer,
BrightMarbles
Adam Houghton
Senior Software Developer,
SAS Institute