Development and programming tools are used to build frameworks, and they can be used for creating, debugging, and maintaining programs — and much more. The resources in this Zone cover topics such as compilers, database management systems, code editors, and other software tools and can help ensure engineers are writing clean code.
Argo CD is a continuous delivery (CD) tool that has gained popularity in DevOps for performing application delivery onto Kubernetes. It relies on a deployment method known as GitOps. GitOps is a mechanism that can pull the latest code and application configuration from the last known Git commit version and deploy it directly into Kubernetes resources. The wide adoption and popularity are because Argo CD helps developers manage infrastructure and applications lifecycle in one platform. To understand Argo CD in-depth, we must first understand how continuous delivery works for most application teams and how it differs from GitOps. What Is Continuous Delivery? Continuous delivery (CD) is a software delivery process/framework that enables developers to continuously push their code changes into the production servers without accessing the infrastructure through a push-based pipeline. This process reduces the time a code takes to reach its end users and improves release velocity. It is a repeatable process that enables scaling your application to address the growing demands of end users. How Does Continuous Delivery Work? Continuous delivery is a part of the application delivery lifecycle that deploys the application post-build into a resource. In this context, our infrastructure will be Kubernetes without the use of Argo CD. Alternatively, if we use Jenkins, the process will be as follows: To release a feature upgrade or a bug fix to the end users, a pre-configured continuous integration (CI) pipeline will build a Docker image as per the Docker file.Ops Then, a script will push it to the configured Docker repository. To move this image into Kubernetes, the deployment.YAML file will be updated with the new image tag and name for fetching the latest image from the Docker registry. Challenges With CD Into Kubernetes Before the rise of Kubernetes, CD most applications teams enabled CD with tools like Jenkins and Spinnaker. The architecture of Kubernetes is complex, and these tools could not efficiently deploy into Kubernetes and deploy without errors due to the complexity of Kubernetes architecture. These issues made using tools like Jenkins challenging to work with Kubernetes. Challenges with traditional CD while deploying into Kubernetes: Tools Installation and Management One needs to install tools like Kubectl and Helm. These add to the operational activities. Accessing Kubernetes Clusters One must configure access management in the CD tool to enable authorization to Kubernetes clusters and execute changes. If Kubernetes clusters run on a cloud provider, those credentials must be configured and shared outside the cluster. Security and Operational Challenge The configurations will rise in proportion with the increase in clusters which increases operational overhead. While increased operational overheads may not be challenging, it risks the system's security when cluster credentials get shared with external services and tools. Issues While Scaling Infrastructure Each team needs its own set of Kubernetes cluster credentials so that the users can access that specific application resources in a cluster. While operating Kubernetes at scale with a CD tool like Jenkins deploying into multiple clusters needs reconfiguration again for each new cluster. Lack of Visibility When a CD tool deploys an application into Kubernetes without GitOps or a Kubernetes-native CI/CD pipeline, the tool loses visibility into the deployment post applying the configuration to the deployment.YAML files. Once the kubectl command has been executed team must wait until someone reports an incident. Also, the status of execution remains unclear. The continuous delivery into Kubernetes can be made efficient with Argo CD, which works on the principle of GitOps. Before understanding GitOps, let us understand push vs. pull-based CI/CD in the upcoming section on GitOps. What Is GitOps and How Is It Different From Traditional CD? Traditional CD and GitOps differ on the core principles of push and pull-based deployments. Most CI/CD processes work on a push mechanism, which means things move to their respective destination at the trigger of an event. For example, when a developer has finished writing his code, he must execute a set of commands to move his code into the server for deployment. In a Kubernetes environment, the developer has to configure the clusters using tools like Kubectl and Helm in the pipeline to apply changes. Argo is a CD tool that uses a pull-based mechanism. A pull-based CD mechanism means the destination triggers an event to pull the data from the source (Git) to deploy at the destination. Argo CD, which resided inside the cluster for reasons explained later on the blog, pulls the most recent verified version of code into the cluster for deployment. There are a lot of benefits to this model, like improved security and ease of use. This pull-based mechanism is called GitOps, where the source code management systems like Git are treated as the only source of truth for application and configuration data. How Does Argo CD Work? Argo CD works in a reversed flow mechanism as compared to push-style deployments. The new mechanism enables Argo CD to run from inside a Kubernetes cluster. Kubernetes faces challenges with the traditional CD mechanism because CI/CD tools, like Jenkins, sit outside the cluster, whereas Argo CD sits inside the cluster. While inside the cluster, Argo CD pulls changes from Git and applies them to the residing cluster. Instead of pushing changes like older generation tools by being inside the cluster, Argo CD prevents sensitive information from being exposed to other tools outside the Kubernetes cluster and environment. Argo CD can be set up in two simple steps: Deploy the Argo CD agent to the cluster. Configure Argo CD to track a Git repository for changes. When Argo CD monitors change, it automatically applies them to the Kubernetes cluster. When developers commit the new code updates to the Git repository, automated CI pipelines will auto-start the build process and build the container image. Then as per the configurations, the CI pipeline will push and update the Kubernetes manifest files. The pipelines will update the new image version name and details on the deployment.yaml file. Argo CD can track this new update, pull the image, and deploy it onto the target cluster. When the Kubernetes cluster is ready, Argo CD sends a report about the status of the application and that the synchronization is complete and correct. Argo CD also works in the other direction, monitoring changes in the Kubernetes cluster and discarding them if they don’t match the current configuration in Git. Best Practices To Follow While Using Argo CD Separate Git repositories for application source code and app configuration Separate Git repo for system configurations Why Separate Repositories? The main reason for having separate repositories for application source code and app configurations is because app config code is not only present in the deployment file but also in the configmaps, secrets, storage, svc, etc. Kubernetes uses. These files change independently from the source code. When a Developer or DevOps wants to change a service.yaml file, which is an application configuration and not a part of the software code, he has to run the whole CI pipeline to sync those changes into production. The CI pipeline will run only when a code change is updated. Clubbing the app configurations and software code together makes the setup complex and inefficient. So as soon as DevOps changes the config on the git repository, Argo CD will become aware of the changes and update the destination cluster as it constantly monitors the repo as soon as config files change in the Git repository. Benefits of Using Argo CD K8s configurations can be defined as code in the Git repository. Config files are not applied for individual laptops of developers/DevOps. Updates are traceable as tags, branches, or pinned specific manifest versions at Git commits. Same and only interface for updating the cluster Git as the single source of truth Avoid untraceable kubectl command applications Version-controlled changes with audit trail Single sign-on (SSO) with providers such as GitLab, GitHub, Microsoft, OAuth2, OIDC, LinkedIn, LDAP, and SAML 2.0 Support for webhooks triggering actions in GitLab, GitHub, and BitBucket Easy Rollbacks If a new code commit is pushed into the Git repository and the changes are applied to the cluster with Argo CD auto sync, the cluster fails. DevOps can revert to the previous working state from the list of the last known best repository versions, just like a Windows restore point. Also, one need not process the laborious process of manually riveting every cluster and doing a clean-up, as Argo CD will do that all by itself. Avoid Snowflake Clusters With Argo CD Argo CD is watching the Git repository and changes in the cluster. Anytime a change happens in the Git repository or the cluster, Argo CD will compare the two states to check for any misconfigurations or differences. If they don't match the desired state defined in the Git repo with the actual state of the cluster, Argo CD will become active and quickly sync the cluster as described in the cluster. So, in that case, if someone goes and makes manual changes in the cluster, Argo CD will detect it and sync to the desired state. It overrides the manual changes. The automatic override helps the system stay stable and guarantees that the Git repository is the only source of truth at any point in time. It also provides full transparency of the cluster and lets it become a Snowflake cluster. Recreate Clusters From Scratch When a cluster completely crashes, and one has to build it from scratch, Argo CD can be pointed to the Git repository, where the complete cluster configuration is defined. It will recreate the same state as the previous one. This is a fully autonomous process where developers and DevOps do not have to worry about disaster recovery post-clean-up processes. This is possible because Argo CD accepts the cluster configuration as code in a declarative way. What Does Argo CD Provide Over Standard GitOps Benefits? Better Team Collaboration With Easy Access Control Production clusters must have limited access, and only some of your team members should be allowed access. To configure different access rules to these clusters, Argo CD can enable approvals for pull requests for authorized developers and engineers. This helps in managing the cluster permissions. No need to create a cluster role and user account on Kubernetes. Non-human users like CI/CD tools or other peripheral tools in the DevOps ecosystem outside the cluster can be configured in Argo CD, which resides inside the cluster. This architecture of Argo CD ensures that those credentials remain inside the cluster making the system robust and secure. Argo CD Architecture Overview Argo CD is a Kubernetes-native CD tool that supports and reads various Kubernetes manifest files such as YAML, Ksonnet, Jsonnet, Helm charts, and Kustomize. It can follow updates to branches, tags, or pinned to a specific version of manifests at a Git commit. Argo CD control plane consists of three main components: Application Controller API Server Repository Service Application Controller Argo CD can detect changes in Git and sync with the repo due to the Application Controller component. This feature enables syncing out-of-date or modified destination configurations to the last approved Git version. The application controller syncs between the local cache created by the repo service and the Git repository because it is a less resource-intensive process. The application controller can also be configured to accept direct changes in code and configuration at the destination without reverting back to the last known configuration on Git. This authority must be granted only to a selected team resource. When this direct change is made, it notifies the DevOps and developers about the difference in the configuration to update the Git repositories. API Server The API server, like a Kubernetes API server, is a service to expose the components of Kubernetes and Argo CD to interfaces like a CLI or web GUI or other third-party tools. The APIs are primarily used to carry out functionalities like application deployment and management, executing rollback of any user-defined actions, managing cluster credentials stored in K8s secrets, and enforcing RBAC Git webhooks. Repository Service Accessing the Git repository is always time-consuming for Argo CD; accessing the Git repo every time will constitute a pull request. Hence, this internal Repository Service makes a local cache of the application manifest and Git repositories on the Kubernetes cluster a replica of the Git repository. It is responsible for generating and returning Kubernetes manifests on input data like repository URL, Git revisions (i.e., branch, tags), application path, and template-specific settings; the server generates Kubernetes manifests.
Fargate vs. Lambda has recently been a trending topic in the serverless space. Fargate and Lambda are two popular serverless computing options available within the AWS ecosystem. While both tools offer serverless computing, they differ regarding use cases, operational boundaries, runtime resource allocations, price, and performance. This blog aims to take a deeper look into the Fargate vs. Lambda battle. What Is AWS Fargate? AWS Fargate is a serverless computing engine offered by Amazon that enables you to efficiently manage containers without the hassles of provisioning servers and the underlying infrastructure. When cluster capacity management, infrastructure management, patching, and provisioning resource tasks are removed, you can finally focus on delivering faster and better quality applications. AWS Fargate works with Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS), supporting a range of container use cases such as machine learning applications, microservices architecture apps, on-premise app migration to the cloud, and batch processing tasks. Without AWS Fargate Developers build container images Define EC2 instances and deploy them Provision memory and compute resources and manage them Create separate VMs to isolate applications Run and manage applications Run and manage the infrastructure Pay EC2 instances usage charges When AWS Fargate Is Implemented Developers build container images Define compute and memory resources Run and manage apps Pay compute resource usage charges In the Fargate vs. Lambda context, Fargate is the serverless compute option in AWS used when you already have containers for your application and simply want to orchestrate them easier and faster. It works with Elastic Kubernetes Service (EKS) as well as Elastic Container Service (ECS). EKS and ECS have two types of computing options: 1. EC2 type: With this option, you need to deal with the complexity of configuring Instances/Servers. This can be a challenge for inexperienced users. You must set up your EC2 instances and put containers inside the servers with some help from the ECS or EKS configurations. 2. Fargate type: This option allows you to reduce the server management burden while easily updating and increasing the configuration limits required to run Fargate. What Is Serverless? Before delving deep into the serverless computing battle of Lambda vs. Fargate or Fargate vs. Lambda, it’s important first to gain a basic understanding of the serverless concept. Serverless computing is a technology that enables developers to run applications without needing to provision server infrastructure. The cloud provider will provide the backend infrastructure on-demand and charge you according to a pay-as-you-go model. The term “serverless” might be misleading for some people. Indeed, it’s important to note that serverless technology doesn’t imply the absence of servers. Rather, the cloud provider will manage the server infrastructure with this technology, allowing developers to concentrate their efforts on an app’s front-end code and logic. Resources are spun when the code executes a function and terminates when the function stops. Billing is based on the duration of the execution time of the resources. Therefore, operational costs are optimized because you don’t pay for idle resources. With serverless technology, you can say goodbye to capacity planning, administrative burdens, and maintenance. Furthermore, you can enjoy high availability and disaster recovery at zero cost. Auto-scaling to zero is also available. Finally, resource utilization is 100%, and billing is done granularly, measuring 100 milliseconds as a unit. What Is AWS Lambda? AWS Lambda is an event-driven serverless computing service. Lambda runs predefined code in response to an event or action, enabling developers to perform serverless computing. This cross-platform was developed by Amazon and first released in 2014. It supports major programming languages such as C#, Python, Java, Ruby, Go, and Node.js. It also supports custom runtime. Some of the popular use cases of Lambda include updating a DynamoDB table, uploading data to S3 buckets, and running events in response to IoT sensor data. The pricing is based on milliseconds of usage, rounding off to the nearest millisecond. Moreover, Lambda allows you to manage Docker containers of sizes up to 50 GB via ECR. When you compare Fargate vs. Lambda, Fargate is for containerized applications running for days, weeks, or years. Lambda is designed specifically to handle small portions of an application, such as a function. For instance, a function that clears the cache every 6 hours and lasts for 30 seconds can be executed using Lambda. A Typical AWS Lambda Architecture AWS Lambda is a Function-as-a-Service (FaaS) that helps developers build event-driven apps. In the app’s compute layer, Lambda triggers AWS events. What are the three core components of Lambda architecture? 1) Function: A function is a piece of code written by developers to perform a task. The code also contains the details of the runtime environment of the function. The runtime environments are based on Amazon Linux AMI and contain all required libraries and packages. Capacity and maintenance are handled by AWS. a. Code Package: The packaged code containing the binaries and assets required for the code to run. The maximum size is 250 MB or 50 MB in a compressed version. b. Handler: The starting point of the invoked function running a task based on parameters provided by event objects. c. Event Object: A parameter provided to the Handler to perform the logic for an operation. d. Context Object: Facilitates interaction between the function code and the execution environment. The data available for Context Objects include: i. AWS Request ID ii. Remaining time for the function to time out iii. Logging statements to CloudWatch 2) Configuration: Rules that specify how a function is executed. a. IAM Roles: Assigns permissions for functions to interact with AWS services. b. Network Configuration: Specifies rules to run functions inside a VPC or outside a VPC. c. Version: Reverts functions to previous versions. d. Memory Dial: Controls resource allocations to functions. e. Environment Variables: Values injected into the code during the runtime. f. Timeout: Time for a function to run. 3) Event Source: The event that triggers the function. a. Push Model: Functions triggered via S3 objects, API Gateway and Amazon Alexa. b. Pull Model: Lambda pulls events from DynamoDB or Kinesis. A Typical AWS Fargate Architecture What are the four core components of the AWS Fargate architecture? 1) Task Definition: A JSON file that describes definitions for at least one of the application containers. 2) Task: Instantiation of a task definition at a cluster level. 3) Cluster: Tasks or services logically grouped in Amazon ECS. 4) Service: A process that runs tasks in Amazon ECS cluster based on task definitions. Fargate vs Lambda: Performance As far as performance is concerned in the AWS Fargate vs. Lambda debate, AWS Fargate is the winner, as it runs on dedicated resources. Lambda has certain limitations when it comes to allocating computing and memory resources. Based on the selected amount of RAM, AWS allocates the corresponding CPU resources meaning that the user cannot customize CPU resources. Moreover, the maximum available memory for Lambda functions is 10 GB, whereas Fargate allows for 120 GB of memory. Furthermore, Fargate allows you to choose up to 16 vCPU resources. Another notable issue is that a Lambda function only has a run time of 15 minutes for every invocation. On the other hand, in the absence of runtime limitations, the Fargate environment is always in a warm state. Fargate functions must be packaged into containers, increasing the load time to around 60 seconds. This is a very long time compared to Lambda functions which can get started within 5 seconds. Fargate allows you to launch 20 tasks per second using ECS RunTask API. Moreover, you can launch 500 tasks per service in 120 seconds with ECS Service Scheduler. That said, scaling the environment during unexpected spike requests and health monitoring tends to cause a bit of a delay in start-up time. Lambda Cold Starts When Lambda receives a request to execute a task, it starts by downloading the code from S3 buckets and creating an execution environment based on the predefined memory and its corresponding compute resources. If there is any initialization code, Lambda runs it outside the environment and then runs the handler code. The time required for downloading the code and preparing the execution environment is counted as the cold start duration. After executing the code, Lambda freezes the environment so that the same function can run quickly if invoked again. If you run the function concurrently, each invocation gets a cold start. There will also be a code start if the code is updated. The typical time for cold starts falls between 100 ms and 1 second. In light of the foregoing, Lambda falls short in the Lambda vs. Fargate race regarding cold starts. However, Provisioned Concurrency is a solution to reduce cold starts. The runtime choice will also have an impact on Lambda cold starts. For instance, Java runtime involves multiple resources to run the JVM environment, which delays the start. On the other hand, C# or Node.js runtime environments offer lower latencies. Fargate Cold Starts Fargate takes time to provision resources and starts a task. Once the environment is up and running, containers get dedicated resources and run the code as defined. Fargate vs. Lambda: Support AWS Fargate works as an operational layer of a serverless computing architecture to manage Docker-based ECS or Kubernetes-based EKS environments. For ECS, you can define container tasks in text files using JSON. There is support for other runtime environments as well. Fargate offers more capacity deployment control than Lambda, as Lambda is limited to 10GB of space and 10GB of package size for container images and 250 MB for deployments to S3 buckets. Lambda supports all major programming languages, such as Python, Go, Ruby, PHP, C#, Node.js, and Java, and code compilation tools, such as Maven and Gradle. That said, Lambda only supports Linux-based container images. With Fargate, you can develop Docker container images locally using Docker Compose and run them in Fargate without worrying about compatibility issues. Since development and architecture is independent of Fargate, it outperforms Lambda in this particular category. When more control over the container environment is the key requirement, AWS Fargate is definitely the right choice. Fargate vs. Lambda: Costs When comparing Fargate vs. Lambda costs, it is important to note that both tools serve different purposes. While Lambda is a Function-as-a-Service, Fargate is a serverless computing tool for container-based workloads. Lambda costs are billed in milliseconds. AWS Lambda charges $0.20 per 1 Million requests with $0.0000166667 for every GB-second duration for the first 6 Billion GB-seconds / month. The duration costs vary based on the allocated memory. For instance, 128 MB memory costs you $0.0000000021 per ms, and 10 GB memory costs you $0.0000001667 per ms. For example, consider 10 GB of memory with 6 vCPU and concurrency, which is always running. The monthly cost for the foregoing would be $432.50. If the concurrency is two, the price is doubled. If the environment runs half the day, the price gets divided by two. If it’s running for 10 minutes per day, the cost would be $9.10 per month. If you consider the same configuration in Fargate, the prices are drastically lower. Fargate charges a flat rate of $0.04048 per vCPU per hour ($29.145 per month) $0.004445 per GB per hour ($3.20 per month) So, a 10 GB memory with 6 vCPUs running continuously for a month with concurrency one would cost $206.87. Moreover, Fargate separates CPUs from memory, allowing you to choose the right-sized configuration. Therefore, you can save costs by reducing the CPUs depending on your needs. When you consider a concurrency of 10, the difference increases exponentially. Another advantage of Fargate is the spot pricing which offers an additional 30% savings. Notice that Lambda costs are lower than Fargate when the idle time is greater. In light of the foregoing, we can conclude that Lambda is more suitable for workloads that are idle for long periods. Lambda is cost-effective if the resources are idle for a quarter or less of the time. Lambda is the best choice to scale fast or isolate security from an app code. Contrastingly, Fargate suits cloud environments with minimally idle workloads. We think the best option is to implement Infrastructure as Code (IaC) and begin with Lambda. When workloads increase, you can seamlessly switch to Fargate. Fargate vs. Lambda: Easy to Work Lambda is easy to set up and operate as there are minimal knobs to adjust compared to Fargate. More abstraction implies less operational burden. However, it also implies limited flexibility. Lambda comes with a rich ecosystem that offers fully automated administration. You can use the management console or the API to call and control functions synchronously or asynchronously, including concurrency. The runtime supports a common set of functionalities and allows you to switch between different frameworks and languages. As far as operational burdens go, Lambda is easier compared to EC2. Fargate stands between Lambda and EC2 in this category, leaning closer towards Lambda. That said, EC2 offers more flexibility in configuring and operating the environment, followed by Fargate and Lambda. Fargate vs Lambda: Community Both AWS Fargate and Lambda are a part of the AWS serverless ecosystem. As such, both tools enjoy the same level of community support. Both services offer adequate support for new and advanced users, from documentation and how-to guides to tutorials and FAQs. Fargate vs Lambda: Cloud Agnostic Each cloud vendor manages serverless environments differently. For instance, C# functions written for AWS will not work on the Google Cloud. In light of the foregoing, developers need to consider cloud-agnostic issues if multi-cloud and hybrid-cloud architectures are involved. Moving between different cloud vendors involves considerable expenses and operational impacts. As such, vendor lock-in is a big challenge for serverless functions. To overcome this, we suggest using an open-source serverless framework offered by Serverless Inc. Moreover, implementing hexagonal architecture is a good idea because it allows you to move code between different serverless cloud environments. Fargate vs Lambda: Scalability In terms of Lambda vs Fargate scalability, Lambda is known as one of the best scaling technologies available in today’s market. Rapid scaling and scaling to zero are the two key strengths of Lambda. The tool instantly scales from zero to thousands and scales down from 1000 to 0, making it a good choice for low workloads, test environments, and workloads with unexpected traffic spikes. As far as Fargate is concerned, container scaling depends on resizing the underlying clusters. Furthermore, it doesn’t natively scale down to zero. Therefore, you’ll have to shut down Fargate tasks outside business hours to save on operational costs. Tasks such as configuring auto-scaling and updating base container images add up when it comes to maintenance. Fargate vs. Lambda: Security Lambda and Fargate are inherently secure as part of the AWS ecosystem. You can secure the environment using the AWS Identity and Access Management (IAM) service. Similarly, both tools abstract away the underlying infrastructure, which means the security of the infrastructure is managed by other services. The difference between the two tools lies in the IAM configuration. Lambda allows you to customize IAM roles for each function or service, while Fargate customizes each container and pod. Fargate tasks run in an isolated computing environment wherein CPU or memory is not shared with other tasks. Similarly, Lambda functions run in a dedicated execution environment. Also, Fargate offers more control over the environment and more secure touchpoints than Lambda. When to Use Fargate or Lambda? AWS Lambda Use Cases: Operating serverless websites Massively scaling operations Real-time processing of high volumes of data Predictive page rendering Scheduled events for every task and data backup Parse user input and cleanup backend data to increase a website’s rapid response time Analyzing log data on-demand Integrating with external services Converting documents into the user-requested format on-demand Real-Life Lambda Use Cases Serverless Websites: Bustle One of the best use cases for Lambda is operating serverless websites. By hosting frontend apps on S3 buckets and using CloudFront content delivery, organizations can manage static websites and take advantage of the Lambda pricing model. Bustle is a news, entertainment, and fashion website for women. The company was having difficulties scaling its application. In addition, server management, monitoring, and automation was becoming an important administrative burden. The company, therefore, decided to move to AWS Lambda with API Gateway and Amazon Kinesis to run serverless websites. Now, the company doesn’t have to worry about scaling, and its developers can deploy code at an extremely low cost. Event-driven Model for Workloads With Idle Times: Thomson Reuters Companies that manage workloads that are idle most of the time can benefit from the Lambda serverless feature. A notable example is Thomson Reuters, one of the world’s most trusted news organizations. The company wanted to build its own analytics engine. The small team working on this project desired a lessened administrative burden. At the same time, the tool needed to scale elastically during breaking news. Reuters chose Lambda. The tool receives data from Amazon Kinesis and automatically loads this data in a master dataset in an S3 bucket. Lambda is triggered with data integrations with Kinesis and S3. As such, Reuters enjoyed high scalability at the lowest cost possible. Highly Scalable Real-time Processing Environment: Realtor.com AWS Lambda enables organizations to scale resources while instantly cost-effectively processing tasks in real-time. Realtor.com is a leader in the real estate market. After the move to the digital world, the company started experiencing exponential traffic growth. Furthermore, the company needed a solution to update ad listings in real-time. Realtor.com chose AWS for its cloud operations. The company uses Amazon Kinesis Data Streams to collect and stream ad impressions. The internal billing system consumes this data using Amazon Kinesis Firehose, and the aggregate data is sent to the Amazon Redshift data warehouse for analysis. The application uses AWS Lambda to read Kinesis Data Streams and process each event. Realtor.com is now able to massively scale operations cost-effectively while making changes to ad listings in real-time. AWS Fargate Use Cases AWS Fargate is the best choice for managing container-based workloads with minimal idle times. Build, run, and manage APIs, microservices, and applications using containers to enjoy speed and immutability Highly scalable container-based data processing workloads Migrate legacy apps running on EC2 instances without refactoring or rearchitecting them Build and manage highly scalable AI and ML development environments Real-Life Use Cases Samsung Samsung is a leader in the electronics category. The company operates an online portal called “Samsung Developers,” which consists of SmartThings Portal for the Internet of Things (IoT), Bixby Portal for voice-based control of mobile services, and Rich Communication Services (RCS) for mobile messaging. The company was using Amazon ECS to manage the online portal. After the re: Invent 2017 event, Samsung was inspired to implement Fargate for operational efficiency. After migrating to AWS Fargate, the company no longer needed dedicated operators and administrators to manage the web services of the portal. Now, geographically distributed teams simply have to create new container images uploaded to ECR and moved to the test environment on Fargate. Developers can therefore focus more on code, and frequent deployments and administrators can focus more on performance and security. Compute costs were downsized by 44.5%. Quola Insurtech Startup Quola is a Jakarta-based insurance technology startup. The company developed software that automates claim processing using AI and ML algorithms to eliminate manual physical reviews. Quola chose AWS cloud and Fargate to run and manage container-based workloads. Amazon Simple Queue Service (SQS) is used for the message-queuing service. With Fargate, Quola is able to scale apps seamlessly. When a new partner joined the network, data transactions increased from 10,000 to 100,000 in a single day. Nevertheless, the app was able to scale instantly without performance being affected. Vanguard Financial Services Vanguard is a leading provider of financial services in the US. The company moved its on-premise operations to the AWS cloud in 2015 and now manages 1000 apps that run on microservices architecture. With security being a key requirement in the financial industry, Vanguard operates in the secure environment of Fargate. With Fargate, the company could offer seamless computing capacity to its containers and reduce costs by 50%. Considerations When Moving to a Serverless Architecture Inspired by the amazing benefits of serverless architecture, many businesses are aggressively embracing the serverless computing model. Here are the steps to migrate monolith and legacy apps to a serverless architecture. a) Monolith to Microservices: Most legacy apps are built using a monolith architecture. When such is the case, the first step is to break the large into smaller and modular microservices, after which each microservice will perform a specific task or function. b) Implement each Microservice as a REST API: The next step is identifying the best fit within these microservices. Implement each microservice as a REST API with API endpoints as resources. Amazon API Gateway is a fully managed service that can help you. c) Implement a Serverless Compute Engine: Implement a serverless compute engine such as Lambda or Fargate and move the business logic to the serverless tool such that AWS provisions resources every time a function is invoked. d) Staggered Deployment Strategy: Migrating microservices to the serverless architecture can be done in a staggered process. Identify the right services and then build, test, and deploy them. Continue this process to smoothly and seamlessly move the entire application to the new architecture. Considerations for Moving to Amazon Lambda Migrating legacy apps to Lambda is not a difficult job. If your application is written in any Lambda-supported languages, you can simply refactor the code and migrate the app to Lambda. You simply need to make some fundamental changes, such as changing the dependency on local storage to S3 or updating authentication modules. When Fargate vs. Lambda security is considered, Lambda has fewer touchpoints to secure than Fargate. If you are using Java runtime, keep in mind that the size of the runtime environment and resources can result in more cold starts than with Node.js or C#. Another key point to consider is memory allocation. Currently, Lambda’s maximum memory allocation is 3 GB. If your application requires more computing and memory resources, Fargate is a better choice. Considerations for Moving to AWS Fargate While AWS manages resource provisioning, customers still need to handle network security tasks. For instance, when a task is created, AWS creates an Elastic Network Interface (ENI) in the VPC and automatically attaches each task ENI to its corresponding subnet. Therefore, managing the connectivity between the ENI and its touch points is the customer’s sole responsibility. More specifically, you need to manage ENI access to AWS EC2, CloudWatch, Apps running on-premise or other regions, Egress, Ingress, etc. Moreover, audit and compliance aspects must be carefully managed, which is why Fargate is not preferred for highly regulated environments. Conclusion The Fargate vs Lambda battle is getting more and more interesting as the gap between container-based and serverless systems is getting smaller with every passing day. There is no silver bullet when deciding which service is the best. With the new ability to deploy Lambda functions as Docker container images, more organizations seem to lean towards Lambda. On the other hand, organizations that need more control over the container runtime environment are sticking with Fargate.
Secrets management in Docker is a critical security concern for any business. When using Docker containers, it is essential to keep sensitive data, such as passwords, API keys, and other credentials, secure. This article will discuss some best practices for managing secrets in Docker, including how to store them securely and minimize their exposure. We will explore multiple solutions: using Docker Secrets with Docker Swarm, Docker Compose, or Mozilla SOPS. Feel free to choose what’s more appropriate to your use case. But most importantly is to remember to never hard-code your Docker secrets in plain text in your Dockerfile! Following these guidelines ensures your organization’s sensitive information remains safe even when running containerized services. 4 Ways To Store and Manage Secrets in Docker 1. Using Docker Secrets and Docker Swarm Docker Secrets and Docker Swarm are two official and complimentary tools allowed to securely manage secrets when running containerized services. Docker Secrets provides a secure mechanism for storing and retrieving secrets from the system without exposing them in plain text. It enables users to keep their credentials safe by encrypting the data with a unique key before passing it to the system. Docker Swarm is a powerful tool for managing clusters of nodes for distributed applications. It provides an effective means of deploying containerized applications at scale. With this tool, you can easily manage multiple nodes within a cluster and automatically distribute workloads among them. This helps ensure your application has enough resources available at all times, even during peak usage periods or unexpected traffic spikes. Together, these two tools provide an effective way to ensure your organization’s sensitive information remains safe despite ever-evolving security needs. Let’s see how to create and manage an example secret. Creating a Secret To create a secret, we need to first initialize Docker Swarm. You can do so using the following command: docker swarm init Once the service is initialized, we can use the docker secret create command to create the secret: ssh-keygen -t rsa -b 4096 -N "" -f mykey docker secret create my_key mykey rm mykey In these commands, we first create an SSH key using the ssh-keygen command and write it to mykey. Then, we use the Docker secret command to generate the secret. Ensure you delete the mykey file to avoid any security risks. You can use the following command to confirm the secret is created successfully: docker secret ls We can now use this secret in our Docker containers. One way is to pass this secret with –secret flag when creating a service: docker service create --name mongodb --secret my_mongodb_secret redis:latest We can also pass this secret to the docker-compose.yml file. Let’s take a look at an example file: version: '3.7' services: myapp: image: mydummyapp:latest secrets: - my_secret volumes: - type: bind source: my_secret_key target: /run/secrets/my_secret read_only: true secrets: my_secret: external: true In the example compose file, the secrets section defines a secret named my_secret_key (discussed earlier). The myapp service definition specifies that it requires my_secret_key , and mounts it as a file at /run/secrets/my_secret in the container. 2. Using Docker Compose Docker Compose is a powerful tool for defining and running multi-container applications with Docker. A stack is defined by a docker-compose file allowing you to define and configure the services that make up your application, including their environment variables, networks, ports, and volumes. With Docker Compose, it is easy to set up an application in a single configuration file and deploy it quickly and consistently across multiple environments. Docker Compose provides an effective solution for managing secrets for organizations handling sensitive data such as passwords or API keys. You can read your secrets from an external file (like a TXT file). But be careful not to commit this file with your code: version: '3.7' services: myapp: image: myapp:latest secrets: - my_secret secrets: my_secret: file: ./my_secret.txt 3. Using a Sidecar Container A typical strategy for maintaining and storing secrets in a Docker environment is to use sidecar containers. Secrets can be sent to the main application container via the sidecar container, which can also operate a secrets manager or another secure service. Let’s understand this using a Hashicorp Vault sidecar for a MongoDB container: First, create a Docker Compose (docker-compose.yml) file with two services: mongo and secrets. In the secrets service, use an image containing your chosen secret management tool, such as a vault. Mount a volume from the secrets container to the mongo container so the mongo container can access the secrets stored in the secrets container. In the mongo service, use environment variables to set the credentials for the MongoDB database, and reference the secrets stored in the mounted volume. Here is the example compose file: version: '3.7' services: mongo: image: mongo volumes: - secrets:/run/secrets environment: MONGO_INITDB_ROOT_USERNAME_FILE: /run/secrets/mongo-root-username MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo-root-password secrets: image: vault volumes: - ./secrets:/secrets command: ["vault", "server", "-dev", "-dev-root-token-id=myroot"] ports: - "8200:8200" volumes: secrets: 4. Using Mozilla SOPS Mozilla SOPS (Secrets Ops) is an open-source platform that provides organizations with a secure and automated way to manage encrypted secrets in files. It offers a range of features designed to help teams share secrets in code in a safe and practical way. The following assumes you are already familiar with SOPS, if that’s not the case, start here. Here is an example of how to use SOPS with docker-compose.yml: version: '3.7' services: myapp: image: myapp:latest environment: API_KEY: ${API_KEY} secrets: - mysecrets sops: image: mozilla/sops:latest command: ["sops", "--config", "/secrets/sops.yaml", "--decrypt", "/secrets/mysecrets.enc.yaml"] volumes: - ./secrets:/secrets environment: # Optional: specify the path to your PGP private key if you encrypted the file with PGP SOPS_PGP_PRIVATE_KEY: /secrets/myprivatekey.asc secrets: mysecrets: external: true In the above, the myapp service requires a secret called API_KEY. The secrets section uses a secret called mysecrets, which is expected to be stored in an external key/value store, such as Docker Swarm secrets or HashiCorp Vault. The sops service uses the official SOPS Docker image to decrypt the mysecrets.enc.yaml file, which is stored in the local ./secrets directory. The decrypted secrets are mounted to the myapp service as environment variables. Note: Make sure to create the secrets directory and add the encrypted mysecrets.enc.yaml file and the sops.yaml configuration file (with SOPS configuration) in that directory. Scan for Secrets in Your Docker Images Hard coding secrets in Docker is a significant security risk, making them vulnerable to attackers. We have seen different best practices to avoid hard-coding secrets in plain text in your Docker images, but security doesn’t stop there. You Should Also Scan Your Images for Secrets All Dockerfiles start with a FROM directive that defines the base image. It’s important to understand when you use a base image, especially from a public registry like Docker Hub, you are pulling external code that may contain hardcoded secrets. More information is exposed than visible in your single Dockerfile. Indeed, it’s possible to retrieve a plain text secret hard-coded in a previous layer starting from your image. In fact, many public Docker images are concerned: in 2021, we estimated that **7% of the Docker Hub images contained at least one secret.** Fortunately, you can easily detect them with ggshield (GitGuardian CLI). For example: ggshield secret scan docker ubuntu:22.04 Conclusion Managing secrets in Docker is a crucial part of preserving the security of your containerized apps. Docker includes several built-in tools for maintaining secrets, such as Docker Secrets and Docker Compose files. Additionally, organizations can use third-party solutions, like HashiCorp Vault and Mozilla SOPS, to manage secrets in Docker. These technologies offer extra capabilities, like access control, encryption, and audit logging, to strengthen the security of your secret management. Finally, finding and limiting accidental or unintended exposure of sensitive information is crucial to handling secrets in Docker. Companies are invited to use secret scanning tools, such as GitGuardian, to scan the Docker images built in their CI/CD pipelines as mitigation to prevent supply-chain attacks. If you want to know more about Docker security, we also summarized some of the best practices in a cheat sheet.
Docker has revolutionized the way we build and deploy applications. It provides a platform-independent environment that allows developers to package their applications and dependencies into a single container. This container can then be easily deployed across different environments, making it an ideal solution for building and deploying applications at scale. Building Docker images from scratch is a must skill that any DevOps engineer needs to acquire for working with Docker. It allows you to create custom images tailored to your application's specific needs, making your deployments more efficient and reliable. Here, in this blog, we'll explore Docker images, its benefits, the process of building Docker images from scratch, and the best practices for building a Docker image. What Is a Docker Image? A Docker image is a lightweight, standalone, executable package that includes everything needed to run the software, including code, libraries, system tools, and settings. Docker images are built using a Dockerfile, which is a text file that contains a set of instructions for building the image. These instructions specify the base image to use, the packages and dependencies to install, and the configuration settings for the application. Docker images are designed to be portable and can be run on any system that supports Docker. They are stored in a central registry, such as Docker Hub, where others can easily share and download. By using Docker images, developers can quickly and easily deploy their applications in a consistent and reproducible manner, regardless of the underlying infrastructure. This makes Docker images an essential tool for modern software development and deployment. Benefits of Building a Docker Image By building image Docker, you can improve the consistency, reliability, and security of your applications. In addition, Docker images make it easy to deploy and manage applications, which helps to reduce the time and effort required to maintain your infrastructure. Here are some major benefits of building a Docker image: Portability: Docker images are portable and can run on any platform that supports Docker. This makes moving applications between development, testing, and production environments easy. Consistency: Docker images provide a consistent environment for running applications. This ensures that the application behaves the same way across different environments. Reproducibility: Docker images are reproducible, which means you can recreate the same environment every time you run the image. Scalability: Docker images are designed to be scalable, which means that you can easily spin up multiple instances of an application to handle increased traffic. Security: Docker images provide a secure way to package and distribute applications. They allow you to isolate your application from the host system and other applications running on the same system. Efficiency: Docker images are lightweight and take up minimal disk space. This makes it easy to distribute and deploy applications quickly. Versioning: Docker images can be versioned, which allows you to track changes and roll back to previous versions if necessary. Structure of a Docker Image A Docker image is a read-only template that contains the instructions for creating a Docker container. Before you learn how to build a Docker image, let's read about its structure first. The structure of a Docker image includes the following components: Base Image A Docker image is built on top of a base image, which is the starting point for the image. The base image can be an official image from the Docker Hub registry or a custom image created by another user. Filesystem The filesystem of a Docker image is a series of layers that represent the changes made to the base image. Each layer contains a set of files and directories that represent the differences from the previous layer. Metadata Docker images also include metadata that provides information about the image, such as its name, version, author, and description. This metadata is stored in a file called the manifest. Dockerfile The Dockerfile is a text file that contains the instructions for building the Docker image. It specifies the base image, the commands to run in the image, and any additional configuration needed to create the image. Before learning how to build the docker image using the Docker build command from Dockerfile, knowing how dockerfile works will be helpful. Configuration Files Docker images may also include configuration files that are used to customize the image at runtime. These files can be mounted as volumes in the container to provide configuration data or environment variables. Runtime Environment Finally, Docker images may include a runtime environment that specifies the software and libraries needed to run the application in the container. This can include language runtimes such as Python or Node.js or application servers such as Apache or Nginx. The structure of a Docker image is designed to be modular and flexible, allowing technology teams to create images tailored to their specific needs while maintaining consistency and compatibility across different environments. How to Build a Docker Image? To build a Docker image, you need to follow these steps: Create a Dockerfile A Dockerfile is a script that contains instructions on how to build your Docker image. The Dockerfile specifies the base image, dependencies, and application code that are required to build the image. After creating a Dockerfile and understanding how Dockerfile works, move to the next step. Define the Dockerfile Instructions In the Dockerfile, you need to define the instructions for building the Docker image. These instructions include defining the base image, installing dependencies, copying files, and configuring the application. Build the Docker Image To build a Docker image, you need to use the Docker build command. This command takes the Dockerfile as input and builds the Docker image. After using the Docker build command with Dockerfile, you can also specify the name and tag for the image using the -t option. Test the Docker Image Once the Docker image is built, you can test it locally using the docker run command. This command runs a container from the Docker image and allows you to test the application. Push the Docker Image to a Registry Once you have tested the Docker image, you can push it to a Docker registry such as Docker Hub or a private registry. This makes it easy to share the Docker image with others and deploy it to other environments. Let's see this Docker build command example. Once you've created your Dockerfile, you can use the "docker build" command to build the image. Here's the basic syntax for the docker build command with dockerfile: (php) docker build -t <image-name> <path-to-Dockerfile> Here, in this Docker build command example, if your Dockerfile is located in the current directory and you want to name your image "my-app," you can use the following Docker build command from dockerfile. (perl) docker build -t my-app This Docker builds command builds the Docker image using the current directory as the build context and sets the name and tag of the image to "my-app." Best Practices for Building a Docker Image Here are some best practices to follow when building a Docker image: First, use a small base image: Use a small base image such as Alpine Linux or BusyBox while building an image Docker. This helps to reduce the size of your final Docker image and improves security by minimizing the attack surface. Use a .dockerignore file: Use a .dockerignore file to exclude files and directories that are not needed in the Docker image. This helps to reduce the size of the context sent to the Docker daemon during the build process. Use multistage builds: Use multistage builds to optimize your Docker image size. Multistage builds allow you to build multiple images in a single Dockerfile, which can help reduce the number of layers in your final image. Minimize the number of layers: Minimize the number of layers in your Docker image to reduce the build time and image size. Each layer in a Docker image adds overhead, so it's important to combine multiple commands into a single layer. Use specific tags: Use specific tags for your Docker image instead of the latest tag. This helps to ensure that you have a consistent and reproducible environment. Avoid installing unnecessary packages: Avoid installing unnecessary packages in your Docker image to reduce the image size and improve security. Use COPY instead of ADD: Use the COPY command instead of ADD to copy files into your Docker image. The COPY command is more predictable and has fewer side effects than the ADD command. Avoid using root user: Avoid using the root user in your Docker image to improve security. Instead, create a non-root user and use that user in your Docker image. Docker Images: The Key to Seamless Container Management By following these steps and practices outlined in this blog, you can create custom Docker images tailored to your application's specific needs. This will not only make your deployments more efficient and reliable, but it will also help you to save time and resources. With these skills, you can take your Docker knowledge to the next level and build more efficient and scalable applications. Docker is a powerful tool for building and deploying applications, but it can also be complex and challenging to manage. Whether you're facing issues with image compatibility, security vulnerabilities, or performance problems, it's important to have a plan in place for resolving these issues quickly and effectively.
Let’s start with the basics. After successfully starting a Redpanda or Apache Kafka® cluster, you want to stream data into it right away. No matter what tool and language you chose, you will immediately be asked for a list of bootstrap servers for your client to connect to it. This bootstrap server is just for your client to initiate the connection to one of the brokers in the cluster, and then it will provide your client with initial sets of metadata. The metadata tells the client the currently available brokers, and which the leader of each partition is hosted by which brokers so that the client can initiate a direct connection to all brokers individually. The diagram below will give you a better idea. The client figures out where to stream the data based on the info given. Depending on the number of partitions and where they are hosted, your client will push or pull from the partitions to/from its host brokers. Both the Kafka address and the advertised Kafka address are needed. Kafka Address is used for Kafka brokers to locate each other, and the advertised address for the client to find them. In this post, we’ll help you understand the advertised Kafka address, how to use it in Docker and Kubernetes (K8s), and how to debug it. When To Use Kafka Address and Advertised Kafka Address When starting up your Redpanda cluster, Kafka Address is used to bind the Redpanda service to its host and use the established endpoint to start accepting requests. {LISTENER_NAME}://{HOST_NAME}:{PORT} The broker uses the advertised Kafka address in the metadata, so your client will take the address to locate other brokers. To set it, use --kafka-addr and --advertise-kafka-addr with RPK or kafka_api or advertised_kafka_api inside /etc/redpanda/redpanda.yaml for each broker. Until this point, everything seems straightforward. And, you might start to wonder whether the Kafka address and advertised Kafka address are actually redundant. It starts to get tricky when your client has no visibility into the cluster host, and if you pass the same internal address to the client, it won’t be able to resolve it. So, we need to modify the advertised Kafka address to let the Kafka client understand and be reachable from outside (i.e., external IP). How To Use Kafka Address and Advertised Kafka Address in Docker (Container) Another problem that often comes up is while running Redpanda brokers in Docker (container). The same applies to other more complex network topologies. But fear not, you already know the mechanics. All you need to do is put the right address for clients that reside in different places. When running the Docker container, it creates its own network by default, but in the case where you need to have multiple of them communicating, you will need to set up a network (sometimes even multiple layers of network) by bridging them together. We know that the Kafka address is used for binding to the host, we’ll just use 0.0.0.0 as it will bind to all interfaces in the host and any port of your wish (do not use an already occupied port). An example could be 0.0.0.0:9092 and 0.0.0.0:9095 for each broker running in the Docker container, you will register a name in the network, if your client is trying to access the broker within the network, all you need to do is set the advertised Kafka address to it’s registered hostname in the network. For example, if your first Redpanda container registered its name as Building-A, you can set the advertised Kafka address to Building-A:9092. For clients outside of the Docker network, where they don’t have access to the network’s routing table, the advertised Kafka address will need to be set to the host where the Docker containers are running on, so the client can find it. And don’t forget that you also need to expose the port and associate that with the host. But, what happens if you have clients who both want to access the cluster at the same time? Simple, add multiple listeners! Each listener will return a set of advertised Kafka addresses for clients in a different environment. Here’s a diagram for you. Using Kafka Address and Advertised Kafka Address in K8s Since Kubernetes is a platform that orchestrates containers, the concept is very similar to running Docker containers, but on a larger scale. In a typical Redpanda cluster, you will want to install a single Redpanda broker in an individual worker node. All the pods running in the K8s would get assigned an internal address, that is only visible inside the Kubernetes environment, if the client is running outside of Kubernetes, it will need a way to find the brokers. So you can use NodePort to expose the port and use the public IP address of the hosting worker node. For the Kafka address, as usual, just bind it to the local container. For example, 0.0.0.0:9092 and 0.0.0.0:9095. As for the advertised Kafka address, we will need to set two listeners: one for internal connection, and one for external. For internal clients, we can simply use the generated internal service name, for example, if your service name is set to Building-A, the advertised Kafka address will be internal://Building-A:9092.For the external listener use the hosting worker node’s public IP (or domain name ) with the port exposed in NodePort, where you will be assigned a new port. For example, if your first work node has public IP (Domain) as XXX-Blvd, and the new port assigned is 39092, you can set the advertised Kafka address to external://XXX-Blvd:39092 . How to Debug the Advertised Kafka Address When you are able to connect to your cluster with Redpanda Keep (rpk) and your client throws errors like “ENOTFOUND”. Check if the advertised_kafka_api is correctly set, with an address that can be resolved by your client. Shell > curl localhost:9644/v1/node_config {"advertised_kafka_api":[{"name":"internal","address":"0.0.0.0","port":9092},{"name":"external","address":"192.186.0.3","port":19092}]....} If you are running Docker, find out which port 9644 was exposed from Docker. Shell > docker port redpanda-0 8081/tcp -> 0.0.0.0:18081 9644/tcp -> 0.0.0.0:19644 18082/tcp -> 0.0.0.0:18082 19092/tcp -> 0.0.0.0:19092 And cURL. Shell > curl localhost:19644/v1/node_config {"advertised_kafka_api":[{"name":"internal","address":"redpanda-0","port":9092},{"name":"external","address":"localhost","port":19092}]....} If you are running Kubernetes, find out what is the exposed admin port. Shell > kubectl get svc redpanda-external -n redpanda NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) redpanda-external NodePort 10.100.87.34 <none> 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP 3h53m Shell > kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName -n redpanda NAME NODE redpanda-0 ip-1-168-57-208.xx.compute.internal redpanda-1 ip-1-168-1-231.xx.compute.internal redpanda-2 ip-1-168-83-90.xx.compute.internal Shell >kubectl get nodes -o=custom-columns='NAME:.metadata.name,IP:.status.addresses[?(@.type=="ExternalIP")].address' NAME IP ip-1.us-east-2.compute.internal 3.12.84.230 ip-1.us-east-2.compute.internal 3.144.255.61 ip-1.us-east-2.compute.internal 3.144.144.138 And cURL. Shell > curl 3.12.84.230:31644/v1/node_config {"advertised_kafka_api":[{"name":"internal","address":"redpanda-1.redpanda.redpanda.svc.cluster.local.","port":9093},{"name":"default","address":"3.12.84.230","port":31092}].....} Lastly, check if you are connecting to the correct listener(Port). And you’re all done! Conclusion If you made it this far, you should now have a better understanding of what the advertised Kafka address is and how you can use it in Docker and K8s. To learn more about Redpanda, check out our documentation and browse the Redpanda blog for tutorials and guides on how to easily integrate with Redpanda. For a more hands-on approach, take Redpanda's free Community edition for a test drive!
If you're tired of managing your infrastructure manually, ArgoCD is the perfect tool to streamline your processes and ensure your services are always in sync with your source code. With ArgoCD, any changes made to your version control system will automatically be synced to your organization's dedicated environments, making centralization a breeze. Say goodbye to the headaches of manual infrastructure management and hello to a more efficient and scalable approach with ArgoCD! This post will teach you how to easily install and manage infrastructure services like Prometheus and Grafana with ArgoCD. Our step-by-step guide makes it simple to automate your deployment processes and keep your infrastructure up to date. We will explore the following approaches: Installation of ArgoCD via Helm Install Prometheus via ArgoCD Install Grafana via ArgoCD Import Grafana dashboard Import ArgoCD metrics Fire up an Alert Prerequisites: Helm Kubectl Docker for Mac with Kubernetes Installation of ArgoCD via Helm To install ArgoCD via Helm on a Kubernetes cluster, you need to: Add the ArgoCD Helm chart repository. Update the Helm chart repository. Install the ArgoCD Helm chart using the Helm CLI. Finally, verify that ArgoCD is running by checking the status of its pods. # Create a namespace kubectl create namespace argocd # Add the ArgoCD Helm Chart helm repo add argo https://argoproj.github.io/argo-helm # Install the ArgoCD helm upgrade -i argocd --namespace argocd --set redis.exporter.enabled=true --set redis.metrics.enabled=true --set server.metrics.enabled=true --set controller.metrics.enabled=true argo/argo-cd # Check the status of the pods kubectl get pods -n argocd When installing ArgoCD, we enabled two flags that exposed two sets of ArgoCD metrics: Application Metrics : controller.metrics.enabled=true API Server Metrics : server.metrics.enabled=true To access the installed ArgoCD, you will need to obtain its credentials: Username: admin Password: kubectl -n argocd \ get secret \ argocd-initial-admin-secret \ -o jsonpath="{.data.password}" | base64 -d Link a Primary Repository to ArgoCD ArgoCD uses the Application CRD to manage and deploy applications. When you create an Application CRD, you specify the following: Source reference to the desired state in Git. Destination reference to the target cluster and namespace. ArgoCD uses this information to continuously monitor the Git repository for changes and deploy them to the target environment. Let’s put it into action by applying the changes: cat <<EOF | kubectl apply -f - apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: workshop namespace: argocd spec: destination: namespace: argocd server: https://kubernetes.default.svc project: default source: path: argoCD/ repoURL: https://github.com/naturalett/continuous-delivery targetRevision: main syncPolicy: automated: prune: true selfHeal: true EOF Let’s access the server UI by using the kubectl port forwarding: kubectl port-forward service/argocd-server -n argocd 8080:443 Connect to ArgoCD. Install Prometheus via ArgoCD By installing Prometheus, you will be able to leverage the full stack and take advantage of its features. When you install the full stack, you will get access to the following: Prometheus Grafana dashboard, and more. In our demo, we will apply from the Kube Prometheus Stack the following services: Prometheus Grafana AlertManager The node-exporter will add the separately with its own helm chart while deactivating the pre-installed default that comes with the Kube Prometheus Stack. There are two ways to deploy Prometheus: Option 1: By applying the CRD. Option 2: By using automatic deployment based on the kustomization. In our blog, the installation of Prometheus will happen automatically, which means that Option 2 will be applied automatically. Option 1 — Apply the CRD cat <<EOF | kubectl apply -f - apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: prometheus namespace: argocd spec: destination: name: in-cluster namespace: argocd project: default source: repoURL: https://prometheus-community.github.io/helm-charts targetRevision: 45.6.0 chart: kube-prometheus-stack EOF Option 2 — Define the Installation Declaratively This option has already been applied based on the CRD we deployed earlier in the step of linking a primary repository to ArgoCD. The CRD is responsible for syncing our application.yaml files with the configuration specified in the kustomization. After Prometheus will get deployed, then it exposes its metrics to /metrics. To display these metrics in Grafana, we need to define a Prometheus data source. In addition, we also have additional metrics that we want to display in Grafana, so we’ll need to scrape them in Prometheus. Access the Prometheus server UI Let’s access Prometheus by using the kubectl port forwarding: kubectl port-forward service/kube-prometheus-stack-prometheus -n argocd 9090:9090 Connect to Prometheus. Prometheus Node Exporter For the installation of the Node Exporter, we utilized the declarative approach, which also happened in Option 2. The installation process will happen automatically, just like it occurred in Option 2, once we link the primary repository to ArgoCD. We will specify the configuration for the Node Exporter’s application using a declarative approach. Prometheus Operator CRDs Due to an issue with the Prometheus Operator Custom Resource Definitions, we have decided to deploy the CRD separately. The installation process will be automatic, similar to the one in Option 2, which relied on linking a primary repository to ArgoCD in an earlier step. Install Grafana via ArgoCD We used the same declarative approach as Option 2 to define the installation of the Grafana. The installation process will take place automatically, just like it does in Option 2, following the earlier step of linking a primary repository to ArgoCD. Since the Grafana installation is part of the Prometheus stack, it was installed automatically when the Prometheus stack was installed. To access the installed Grafana, you will need to obtain its credentials: Username: admin Password: kubectl get secret \ -n argocd \ kube-prometheus-stack-grafana \ -o jsonpath="{.data.admin-password}" | base64 --decode ; echo Let’s access Grafana by using the kubectl port forwarding: kubectl port-forward service/kube-prometheus-stack-grafana -n argocd 9092:80 Connect to Grafana. Importing the ArgoCD Metrics Dashboard Into Grafana We generated a configMap for the ArgoCD Dashboard and deployed it through kustomization. During the deployment of Grafana, we linked the configMap to create the Dashboard and then leveraged Prometheus to extract the ArgoCD metrics data for gaining valuable insights into its performance. The ArgoCD dashboard’s metrics were made available as a result of an earlier section in the blog post: --set server.metrics.enabled=true \ --set controller.metrics.enabled=true This enabled us to view and monitor the metrics easily through the dashboard. Confirm the ArgoCD metrics: # Verify if the services exist kubectl get service -n argocd argocd-application-controller-metrics kubectl get service -n argocd argocd-server-metrics # Configure port forwarding to monitor Application Metrics kubectl port-forward service/argocd-application-controller-metrics -n argocd 8082:8082 # Check the Application Metrics http://localhost:8082/metrics # Configure port forwarding to monitor API Server Metrics kubectl port-forward service/argocd-server-metrics -n argocd 8083:8083 # Check the API Server Metrics http://localhost:8083/metrics Fire up an Alert Execute the following script to trigger an alert: curl -LO https://raw.githubusercontent.com/naturalett/continuous-delivery/main/trigger_alert.sh chmod +x trigger_alert.sh ./trigger_alert.sh Let’s access the Alert Manager: kubectl port-forward service/alertmanager-operated -n argocd 9093:9093 Connect to Alert Manager. Confirm that the workshop alert has been triggered: Clean up the Environment By deleting the workshop ApplicationsSet, all the dependencies that were installed as per the defined kustomization will be removed. Delete the ArgoCD installation and any associated dependencies: kubectl delete crd alertmanagerconfigs.monitoring.coreos.com kubectl delete crd alertmanagers.monitoring.coreos.com kubectl delete crd podmonitors.monitoring.coreos.com kubectl delete crd probes.monitoring.coreos.com kubectl delete crd prometheuses.monitoring.coreos.com kubectl delete crd prometheusrules.monitoring.coreos.com kubectl delete crd servicemonitors.monitoring.coreos.com kubectl delete crd thanosrulers.monitoring.coreos.com kubectl delete crd applications.argoproj.io kubectl delete crd applicationsets.argoproj.io kubectl delete crd appprojects.argoproj.io helm del -n argocd argocd Summary Through our learning process, we have developed proficiency in automating infrastructure management and synchronizing our environment with changes to our source code. Specifically, we have learned how to deploy ArgoCD and utilize its ApplicationsSet to deploy a Prometheus stack. Additionally, we have demonstrated the process of extracting service metrics to Prometheus and visualizing them in Grafana, as well as triggering alerts in our monitoring system. For continued learning and access to valuable resources, we encourage you to explore our tutorial examples on Github.
AWS Gateway is a powerful tool for building APIs that scale to meet the demands of modern web and mobile applications. With AWS Gateway, you can create RESTful APIs that expose your data and business logic to developers who can then build rich, interactive applications that consume your API. REST API is an industry standard for building scalable, distributed web applications. With AWS Gateway, you can easily build a REST API that supports both GET and POST methods, as well as complex query parameters. You can also add support for other HTTP methods, such as PUT, DELETE, and HEAD. Using AWS Gateway, you can quickly create APIs that are secure and robust. You can also use it to deploy your code to a production environment with minimal effort. Additionally, AWS Gateway allows for seamless integration with other AWS services, such as S3 and DynamoDB, enabling you to easily add complex functionality to your APIs. Pre-requisites Before building a RESTful API with AWS Gateway, you should have the following in place: Create an AWS account if you don’t have one already. Log in to the AWS Management Console and navigate to the Amazon API Gateway service. Click on "Create API" and select "REST API". Click on "Actions" to define the resource and click "Create Method". Choose the HTTP verb (e.g. GET, POST, PUT, etc.) and click on the checkmark to create the method. In the "Integration type" section, select "Lambda Function" and enter the name of the Lambda function you want to use to handle the API requests. Click on "Save" to create the API. Select Node from the Runtime Dropdown. Code Example Python import json # Example data data = { "items": [ {"id": 1, "name": "Item 1", "price": 10.99}, {"id": 2, "name": "Item 2", "price": 15.99}, {"id": 3, "name": "Item 3", "price": 20.99}, ] } def lambda_handler(event, context): # Determine the HTTP method of the request http_method = event["httpMethod"] # Handle GET request if http_method == "GET": # Return the data in the response response = { "statusCode": 200, "body": json.dumps(data) } return response # Handle POST request elif http_method == "POST": # Retrieve the request's body and parse it as JSON body = json.loads(event["body"]) # Add the received data to the example data data["items"].append(body) # Return the updated data in the response response = { "statusCode": 200, "body": json.dumps(data) } return response # Handle PUT request elif http_method == "PUT": # Retrieve the request's body and parse it as JSON body = json.loads(event["body"]) # Update the example data with the received data for item in data["items"]: if item["id"] == body["id"]: item.update(body) break # Return the updated data in the response response = { "statusCode": 200, "body": json.dumps(data) } return response # Handle DELETE request elif http_method == "DELETE": # Retrieve the request's body and parse it as JSON body = json.loads(event["body"]) # Find the item with the specified id in the example data for i, item in enumerate(data["items"]): if item["id"] == body["id"]: # Remove the item from the example data del data["items"][i] break # Return the updated data in the response response = { "statusCode": 200, "body": json.dumps(data) } return response else: # Return an error message for unsupported methods response = { "statusCode": 405, "body": json.dumps({"error": "Method not allowed"}) } return response This code defines a Lambda function, lambda_handler, that handles different types of HTTP requests (GET, POST, PUT, DELETE) on some data. The data is an object containing an array of items, each item has an id, name, and price. When the function is called, it first determines the HTTP method of the request from the event object. Then it handles the request accordingly: GET: returns the data in the response with a status code of 200. POST: retrieves the request's body and parses it as JSON, then add the received data to the example data, then returns the updated data in the response with a status code of 200. PUT: retrieves the request's body and parses it as JSON, then updates the example data with the received data, then returns the updated data in the response with a status code of 200. DELETE: retrieves the request's body and parses it as JSON, then find the item with the specified id in the example data and removes it, then returns the updated data in the response with a status code of 200. If the method is not supported, it will return an error message with a status code of 405. Deploy the API by clicking on "Actions" and selecting "Deploy API". Select a deployment stage (e.g. "prod" or "test") and click on "Deploy".Use the generated API endpoint to make requests to your API. Running and Testing the Code in Postman Now, our API is up and running. You can send a test HTTP request through Postman. By sending a request to your invoke URL, you should see a 200 OK status code. For this test, no request body is needed for the incoming request.
Apache Kafka is an event streaming platform that was developed by LinkedIn and later made open-source under the Apache Software Foundation. Its primary function is to handle high-volume real-time data streams and provide a scalable and fault-tolerant architecture for creating data pipelines, streaming applications, and microservices. Kafka employs a publish-subscribe messaging model, in which data is sorted into topics, and publishers send messages to those topics. Subscribers can then receive those messages in real time. The platform offers a scalable and fault-tolerant architecture by spreading data across multiple nodes and replicating data across multiple brokers. This guarantees that data is consistently available, even if a node fails. Kafka's architecture is based on several essential components, including brokers, producers, consumers, and topics. Brokers manage the message queues and handle message persistence, while producers and consumers are responsible for publishing and subscribing to Kafka topics, respectively. Topics function as the communication channels through which messages are sent and received. Kafka also provides an extensive range of APIs and tools to manage data streams and build real-time applications. Kafka Connect, one of its most popular tools and APIs, enables the creation of data pipelines that integrate with other systems. Kafka Streams, on the other hand, allows developers to build streaming applications using a high-level API. In summary, Kafka is a robust and adaptable platform that can be used to construct real-time data pipelines and streaming applications. It has been widely adopted in various sectors, including finance, healthcare, e-commerce, and more. To create a Kafka data stream using Camel, you can use the Camel-Kafka component, which is already included in Apache Camel. Below are the steps to follow for creating a Kafka data stream using Camel: Prepare a Kafka broker and create a topic for the data stream. Set up a new Camel project on your IDE and include the required Camel dependencies, including the Camel-Kafka component. Create a new Camel route within your project that defines the data stream. The route should use the Kafka component and specify the topic to which the data should be sent or received. Select the appropriate data format for the data stream. For instance, if you want to send JSON data, use the Jackson data format to serialize and deserialize the data. Launch the Camel context and the Kafka producer or consumer to start sending or receiving data. Overall, using the Camel-Kafka component with Apache Camel is a simple way to create data streams between applications and a Kafka cluster. Here is the code for reading Table form DB and writing to Kafka cluster: Apache Camel Producer Application: Java import org.apache.camel.builder.RouteBuilder; import org.apache.camel.component.kafka.KafkaConstants; import org.springframework.stereotype.Component; @Component public class OracleDBToKafkaRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { // Configure Oracle DB endpoint String oracleDBEndpoint = "jdbc:oracle:thin:@localhost:1521:orcl"; String oracleDBUser = "username"; String oracleDBPassword = "password"; String oracleDBTable = "mytable"; String selectQuery = "SELECT * FROM " + oracleDBTable; // Configure Kafka endpoint String kafkaEndpoint = "kafka:my-topic?brokers=localhost:9092"; String kafkaSerializer = "org.apache.kafka.common.serialization.StringSerializer"; from("timer:oracleDBPoller?period=5000") // Read from Oracle DB .to("jdbc:" + oracleDBEndpoint + "?user=" + oracleDBUser + "&password=" + oracleDBPassword) .setBody(simple(selectQuery)) .split(body()) // Serialize to Kafka .setHeader(KafkaConstants.KEY, simple("${body.id}")) .marshal().string(kafkaSerializer) .to(kafkaEndpoint); } } Here is the code for reading Kafka Topic and writing the Oracle DB table: Apache Camel Camel Application; Java import org.apache.camel.builder.RouteBuilder; import org.apache.camel.component.kafka.KafkaConstants; import org.springframework.stereotype.Component; @Component public class KafkaToOracleDBRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { // Configure Kafka endpoint String kafkaEndpoint = "kafka:my-topic?brokers=localhost:9092"; String kafkaDeserializer = "org.apache.kafka.common.serialization.StringDeserializer"; // Configure Oracle DB endpoint String oracleDBEndpoint = "jdbc:oracle:thin:@localhost:1521:orcl"; String oracleDBUser = "username"; String oracleDBPassword = "password"; String oracleDBTable = "mytable"; from(kafkaEndpoint) // Deserialize from Kafka .unmarshal().string(kafkaDeserializer) .split(body().tokenize("\n")) // Write to Oracle DB .to("jdbc:" + oracleDBEndpoint + "?user=" + oracleDBUser + "&password=" + oracleDBPassword) .setBody(simple("INSERT INTO " + oracleDBTable + " VALUES(${body})")) .to("jdbc:" + oracleDBEndpoint + "?user=" + oracleDBUser + "&password=" + oracleDBPassword); } }
Docker Swarm: Simplifying Container Orchestration In recent years, containers have become an increasingly popular way to package, distribute, and deploy software applications. They offer several advantages over traditional virtual machines, including faster start-up times, improved resource utilization, and greater flexibility. However, managing containers at scale can be challenging, especially when running large, distributed applications. This is where container orchestration tools come into play, and Docker Swarm is one of the most popular options available. What Is Docker Swarm? Docker Swarm is a container orchestration tool that allows you to deploy and manage a cluster of Docker nodes. Each node is a machine that hosts one or more Docker containers, and together, they form a swarm. Docker Swarm provides a simple and intuitive interface for managing and monitoring your containers, making it an ideal tool for large-scale container deployments. Docker Swarm makes it easy to deploy and manage containerized applications across multiple hosts. It provides features such as load balancing, automatic service discovery, and fault tolerance. With Docker Swarm, you can easily scale your applications up or down by adding or removing Docker nodes from the cluster, making it easy to handle changes in traffic or resource usage. How Does Docker Swarm Work? Docker Swarm allows you to deploy and manage a cluster of Docker nodes. The nodes are machines that host one or more Docker containers, and they work together to form a swarm. When you deploy an application to Docker Swarm, you define a set of services that make up the application. Each service consists of one or more containers that perform a specific function. For example, you might have a service that runs a web server and another service that runs a database. Docker Swarm automatically distributes the containers across the nodes in the swarm, ensuring that each service is running on the appropriate nodes. It also provides load balancing and service discovery, making it easy to access your applications from outside the swarm. Docker Swarm uses a leader-follower model to manage the nodes in the swarm. The leader node is responsible for managing the overall state of the swarm and coordinating the activities of the follower nodes. The follower nodes are responsible for running the containers and executing the tasks assigned to them by the leader node. Docker Swarm is built on top of the Docker Engine, which is the core component of the Docker platform. The Docker Engine runs on each node in the swarm and manages the lifecycle of containers running on that node. When you deploy an application to a Docker Swarm, you define a set of services that make up the application. Each service consists of one or more containers that perform a specific function. For example, you might have a service that runs a web server and another service that runs a database. Docker Swarm automatically distributes the containers across the nodes in the swarm, ensuring that each service is running on the appropriate nodes. It also provides load balancing and service discovery, making it easy to access your applications from outside the swarm. Docker Swarm provides several features that make it easy to manage containers at scale, including: Load Balancing Docker Swarm automatically distributes incoming traffic across the nodes running the containers in the swarm, ensuring that each container receives a fair share of the traffic. Docker Swarm provides built-in load balancing to distribute traffic evenly across containers in a cluster. This helps to ensure that each container receives an equal share of the workload and prevents any single container from becoming overloaded. Automatic Service Discovery Docker Swarm automatically updates a DNS server with the IP addresses of containers running in the swarm. This makes it easy to access your containers using a simple domain name, even as the containers move around the swarm. Docker Swarm automatically assigns unique DNS names to containers, making it easy to discover and connect to services running within the swarm. This feature simplifies the management of large, complex, containerized applications. Fault Tolerance Docker Swarm automatically detects when a container fails and automatically restarts it on another node in the swarm. This ensures that your applications remain available even if individual containers or nodes fail. Scaling Docker Swarm makes it easy to scale your applications up or down by adding or removing nodes from the swarm. This makes it easy to handle changes in traffic or resource usage. Docker Swarm enables easy scaling of containerized applications. As your application traffic grows, you can add more nodes to the cluster, and Docker Swarm automatically distributes the containers across the new nodes. Rolling Updates Docker Swarm allows for rolling updates, where you can update containers without disrupting the application’s availability. This is achieved by updating containers one at a time while other containers continue to handle the traffic. Security Docker Swarm provides built-in security features to help protect your containerized applications. For example, it supports mutual TLS encryption for securing communication between nodes in the cluster. Ease of Use Docker Swarm is designed to be easy to use, with a simple API and command-line interface that makes it easy to deploy and manage containerized applications. High Availability Docker Swarm is designed to provide high availability for containerized applications. It automatically distributes containers across multiple nodes in a cluster and provides fault tolerance so that even if a node or container fails, the application remains available. Overall, Docker Swarm provides a range of powerful features that make it an ideal choice for managing containers at scale. With its support for high availability, scalability, load balancing, service discovery, rolling updates, security, and ease of use, Docker Swarm simplifies the management of containerized applications, allowing you to focus on delivering value to your customers. Benefits of Docker Swarm Docker Swarm offers several benefits for organizations that are deploying containerized applications at scale. These include: Simplified Management Docker Swarm provides a simple and intuitive interface for managing containers at scale. This makes it easy to deploy, monitor, and scale your applications. High Availability Docker Swarm provides built-in fault tolerance, ensuring that your applications remain available even if individual containers or nodes fail. Scalability Docker Swarm makes it easy to scale your applications up or down by adding or removing nodes from the swarm. This makes it easy to handle changes in traffic or resource usage. Compatibility Docker Swarm is fully compatible with the Docker platform, making it easy to use alongside other Docker tools and services. Portability Docker Swarm allows you to easily deploy and manage containerized applications across different environments, including on-premises and in the cloud. This helps to ensure that your applications can be easily moved and scaled as needed, providing flexibility and agility for your business. Conclusion Docker Swarm is a powerful tool for managing containers at scale. It provides a simple and intuitive interface for deploying and managing containerized applications across multiple hosts while also providing features such as load balancing, automatic service discovery, and fault tolerance. Docker Swarm is a very powerful tool for anyone looking to deploy and manage containerized applications at scale. It provides a simple and intuitive interface for managing a cluster of Docker nodes, allowing you to easily deploy and manage services across multiple hosts. With features such as load balancing, service discovery, and fault tolerance, Docker Swarm makes it easy to run containerized applications in production environments. If you’re using Docker for containerization, Docker Swarm is definitely worth checking out.
What Is SIEM? SIEM stands for Security Information and Event Management. It is a software solution that provides real-time analysis of security alerts generated by network hardware and applications. SIEM collects log data from multiple sources such as network devices, servers, and applications, then correlates and analyzes this data to identify security threats. SIEM can help organizations improve their security posture by providing a centralized view of security events across the entire IT infrastructure. It allows security analysts to quickly identify and respond to security incidents and provides detailed reports for compliance purposes. Some of the key features of SIEM solutions include: Log collection and analysis Real-time event correlation and alerting User and entity behavior analytics Threat intelligence integration Compliance reporting SIEM is often used in conjunction with other security solutions, such as firewalls, intrusion detection systems, and antivirus software, to provide comprehensive security monitoring and incident response capabilities. What Is ELK? ELK is an acronym for a set of open-source software tools used for log management and analysis: Elasticsearch, Logstash, and Kibana. Elasticsearch is a distributed search and analytics engine that provides fast search and efficient storage of large volumes of data. It is designed to be scalable and can handle a large number of queries and indexing operations in real-time. Logstash is a data collection and processing tool that allows you to collect logs and other data from multiple sources, such as log files, syslog, and other data sources, and transform and enrich the data before sending it to Elasticsearch. Kibana is a web-based user interface that allows you to visualize and analyze data stored in Elasticsearch. It provides a range of interactive visualizations, such as line graphs, bar charts, and heatmaps, as well as features such as dashboards and alerts. Together, these three tools form a powerful platform for managing and analyzing logs and other types of data, commonly referred to as the ELK stack or Elastic stack. The ELK stack is widely used in IT operations, security monitoring, and business analytics to gain insights from large amounts of data. Ingesting SIEM Data to ELK Ingesting SIEM data into the ELK stack can be useful for organizations that want to combine the security event management capabilities of SIEM with the log management and analysis features of ELK. Here are the high-level steps to ingest SIEM data into ELK: Configure the SIEM to send log data to Logstash, which is part of the ELK stack. Create a Logstash configuration file that defines the input, filters, and output for the SIEM data. Start Logstash and verify that it is receiving and processing SIEM data correctly. Configure Elasticsearch to receive and store the SIEM data. Create Kibana visualizations and dashboards to display the SIEM data. Here is an example of a Logstash configuration file that receives Syslog messages from a SIEM and sends them to Elasticsearch: Python input { syslog { type => "syslog" port => 5514 } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } } } output { elasticsearch { hosts => ["localhost:9200"] index => "siem" } } Once Logstash is configured and running, SIEM data will be ingested into Elasticsearch and can be visualized and analyzed in Kibana. It's important to ensure that the appropriate security measures are in place to protect the SIEM and ELK environments, and to monitor and alert on any security events. Detecting Host Hack Attempt Detecting host hack attempts using SIEM in ELK involves monitoring and analyzing system logs and network traffic to identify suspicious activity that may indicate a hack attempt. Here are the high-level steps to set up host hack attempt detection using SIEM in ELK: Configure the hosts to send system logs and network traffic to a centralized log collection system. Set up Logstash to receive and parse the logs and network traffic data from the hosts. Configure Elasticsearch to store the parsed log data. Use Kibana to analyze the log data and create dashboards and alerts to identify potential hack attempts. Here are some specific techniques that can be used to detect host hack attempts: Monitor for failed login attempts: Look for repeated failed login attempts from a single IP address, which may indicate a brute-force attack. Use Logstash to parse the system logs for failed login events and create a Kibana dashboard or alert to monitor for excessive failed login attempts. Monitor for suspicious network traffic: Look for network traffic to or from known malicious IP addresses or domains. Use Logstash to parse network traffic data and create a Kibana dashboard or alert to monitor for suspicious traffic patterns. Monitor for file system changes: Look for unauthorized changes to system files or settings. Use Logstash to parse file system change events and create a Kibana dashboard or alert to monitor for unauthorized changes. Monitor for suspicious process activity: Look for processes that are running with elevated privileges or that are performing unusual actions. Use Logstash to parse process events and create a Kibana dashboard or alert to monitor for suspicious process activity. By implementing these techniques and regularly monitoring the logs and network traffic, organizations can improve their ability to detect and respond to host hack attempts using SIEM in ELK. Configure Alert in ELK to Detect Host Hack Attempt To configure an alert in ELK to detect a host hack attempt, you can follow these general steps: Create a search query in Kibana that filters logs for Host Hack Attempt events. For example, you can use the following search query to detect failed login attempts: Python from elasticsearch import Elasticsearch es = Elasticsearch() search_query = { "query": { "bool": { "must": [ { "match": { "event.dataset": "auth" } }, { "match": { "event.action": "failed_login" } } ] } } } res = es.search(index="siem", body=search_query) for hit in res['hits']['hits']: print(hit['_source']) Once you have created your search query, save it as a Kibana saved search. Go to the Kibana Alerts and Actions interface and create a new alert. Choose the saved search you created in step 2 as the basis for the alert. Configure the alert to trigger when a certain threshold is met. For example, you can configure the alert to trigger when there are more than 5 failed login attempts within a 5-minute window. Configure the alert to send a notification, such as an email or Slack message, when it triggers. Test the alert to ensure that it is working as expected. Once the alert is configured, it will automatically trigger when it detects a Host Hack Attempt event, such as a failed login attempt. This can help organizations detect and respond to security threats efficiently and effectively. It is important to regularly review and update your alerts to ensure they are detecting the most relevant and important security events. Conclusion Using ELK to detect host hack attempts is an effective approach to enhance the security posture of an organization. ELK provides a powerful combination of log collection, parsing, storage, analysis, and alerting capabilities, which enable organizations to detect and respond to host hack attempts in real-time. By monitoring system logs and network traffic, and using advanced search queries and alerting mechanisms, ELK can help organizations detect a wide range of host hack attempts, including failed login attempts, suspicious network traffic, file system changes, and suspicious process activity. Implementing a robust host hack attempt detection strategy using ELK requires careful planning, configuration, and testing. However, with the right expertise and tools, organizations can create a comprehensive security monitoring system that provides real-time visibility into their network, improves incident response times, and helps prevent security breaches before they occur.
Bartłomiej Żyliński
Software Engineer,
SoftwareMill
Vishnu Vasudevan
Head of Product Engineering & Management,
Opsera
Abhishek Gupta
Principal Developer Advocate,
AWS
Yitaek Hwang
Software Engineer,
NYDIG