DevOps Fundamentals: CI/CD Pipeline Setup with Jenkins and Docker
DevOps practices have revolutionized software delivery by combining development and operations teams, enabling faster releases, improved quality, and enhanced collaboration. This comprehensive guide covers DevOps fundamentals, CI/CD pipeline implementation with Jenkins and Docker, automation strategies, and best practices for building production-ready delivery pipelines.
📑 Table of Contents
- Table of Contents
- 1. DevOps Fundamentals
- What is DevOps?
- CI/CD Pipeline Overview
- 2. Jenkins Installation and Configuration
- Install Jenkins on Ubuntu
- Install Jenkins with Docker
- Initial Jenkins Configuration
- Configure Jenkins Global Tools
- Jenkins Configuration as Code (JCasC)
- 3. Docker Integration
- Install Docker
- Dockerfile Best Practices
- Docker Compose for Multi-Container Setup
- 4. CI/CD Pipeline Setup
- Basic Jenkinsfile
- Advanced Jenkinsfile with Multiple Environments
- 5. Build and Test Automation
- Automated Testing Strategy
- Continuous Testing with Docker
- 6. Deployment Strategies
- Blue-Green Deployment
- Canary Deployment
- Rolling Deployment
- 7. Security and Secrets Management
- Jenkins Credentials Management
- HashiCorp Vault Integration
- Image Security Scanning
- 8. Monitoring and Observability
- Prometheus Metrics in Application
- Jenkins Pipeline Monitoring
- 9. Best Practices
- Pipeline Best Practices
- Docker Best Practices
- Security Best Practices
- Conclusion
- Additional Resources
Table of Contents
- DevOps Fundamentals
- Jenkins Installation and Configuration
- Docker Integration
- CI/CD Pipeline Setup
- Build and Test Automation
- Deployment Strategies
- Security and Secrets Management
- Monitoring and Observability
- Best Practices
1. DevOps Fundamentals
What is DevOps?
DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) to shorten the development lifecycle and provide continuous delivery with high quality.
Key DevOps Principles:
- Automation: Automate repetitive tasks in build, test, and deployment
- Continuous Integration: Frequently merge code changes to detect issues early
- Continuous Delivery: Automate deployment to any environment
- Infrastructure as Code: Manage infrastructure through version-controlled code
- Monitoring and Logging: Continuous feedback from production systems
- Collaboration: Break down silos between development and operations
CI/CD Pipeline Overview
# CI/CD Pipeline Stages
1. Source Control (Git) →
2. Build (Compile, Package) →
3. Test (Unit, Integration, E2E) →
4. Security Scan (SAST, Dependency Check) →
5. Artifact Storage (Docker Registry, Nexus) →
6. Deploy to Staging →
7. Automated Testing →
8. Deploy to Production →
9. Monitor and Alert
2. Jenkins Installation and Configuration
Install Jenkins on Ubuntu
# Update system
sudo apt update && sudo apt upgrade -y
# Install Java (Jenkins requirement)
sudo apt install -y openjdk-11-jdk
# Verify Java installation
java -version
# Add Jenkins repository
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee
/usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]
https://pkg.jenkins.io/debian-stable binary/ | sudo tee
/etc/apt/sources.list.d/jenkins.list > /dev/null
# Install Jenkins
sudo apt update
sudo apt install -y jenkins
# Start Jenkins
sudo systemctl start jenkins
sudo systemctl enable jenkins
# Check status
sudo systemctl status jenkins
# Get initial admin password
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Install Jenkins with Docker
# Create Jenkins home directory
mkdir -p $HOME/jenkins_home
# Run Jenkins container
docker run -d
--name jenkins
-p 8080:8080
-p 50000:50000
-v $HOME/jenkins_home:/var/jenkins_home
-v /var/run/docker.sock:/var/run/docker.sock
jenkins/jenkins:lts
# Get initial admin password
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
# Access Jenkins at http://localhost:8080
Initial Jenkins Configuration
# 1. Access Jenkins web interface
http://your-server-ip:8080
# 2. Enter initial admin password
# 3. Install suggested plugins:
- Git plugin
- Docker plugin
- Pipeline plugin
- Blue Ocean (modern UI)
- Credentials Binding Plugin
- SSH Agent Plugin
# 4. Create admin user
# 5. Configure Jenkins URL
http://your-server-ip:8080/
Configure Jenkins Global Tools
# Navigate to: Manage Jenkins > Global Tool Configuration
# Git Configuration
Name: Default
Path: git (or /usr/bin/git)
# Maven Configuration
Name: Maven-3.9
Install automatically: Yes
Version: 3.9.0
# Docker Configuration
Name: docker
Install automatically: No
Docker installation root: /usr/bin/docker
# JDK Configuration
Name: JDK-11
JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
Jenkins Configuration as Code (JCasC)
# jenkins.yaml
jenkins:
systemMessage: "Jenkins configured automatically by JCasC"
numExecutors: 5
mode: NORMAL
securityRealm:
local:
allowsSignup: false
users:
- id: "admin"
password: "${JENKINS_ADMIN_PASSWORD}"
authorizationStrategy:
loggedInUsersCanDoAnything:
allowAnonymousRead: false
clouds:
- docker:
name: "docker"
dockerApi:
dockerHost:
uri: "unix:///var/run/docker.sock"
templates:
- labelString: "docker-agent"
dockerTemplateBase:
image: "jenkins/agent:latest"
credentials:
system:
domainCredentials:
- credentials:
- usernamePassword:
scope: GLOBAL
id: "docker-hub"
username: "${DOCKER_USERNAME}"
password: "${DOCKER_PASSWORD}"
- string:
scope: GLOBAL
id: "github-token"
secret: "${GITHUB_TOKEN}"
tool:
git:
installations:
- name: "Default"
home: "git"
maven:
installations:
- name: "Maven-3.9"
properties:
- installSource:
installers:
- maven:
id: "3.9.0"
3. Docker Integration
Install Docker
# Install Docker on Ubuntu
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
# Start Docker
sudo systemctl start docker
sudo systemctl enable docker
# Add Jenkins user to docker group
sudo usermod -aG docker jenkins
sudo systemctl restart jenkins
# Verify Docker installation
docker --version
docker compose version
Dockerfile Best Practices
# Multi-stage Dockerfile for Node.js application
# Stage 1: Build
FROM node:18-alpine AS builder
WORKDIR /app
# Copy package files
COPY package*.json ./
# Install dependencies
RUN npm ci --only=production
# Copy application code
COPY . .
# Build application (if needed)
RUN npm run build
# Stage 2: Production
FROM node:18-alpine
# Security: Run as non-root user
RUN addgroup -g 1001 -S nodejs &&
adduser -S nodejs -u 1001
WORKDIR /app
# Copy built artifacts from builder stage
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --from=builder --chown=nodejs:nodejs /app/package*.json ./
# Switch to non-root user
USER nodejs
# Expose port
EXPOSE 3000
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3
CMD node healthcheck.js
# Start application
CMD ["node", "dist/server.js"]
Docker Compose for Multi-Container Setup
# docker-compose.yml
version: '3.8'
services:
app:
build:
context: .
dockerfile: Dockerfile
ports:
- "3000:3000"
environment:
- NODE_ENV=production
- DATABASE_URL=postgresql://postgres:password@db:5432/myapp
- REDIS_URL=redis://redis:6379
depends_on:
db:
condition: service_healthy
redis:
condition: service_started
networks:
- app-network
restart: unless-stopped
db:
image: postgres:15-alpine
environment:
- POSTGRES_DB=myapp
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=password
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 10s
timeout: 5s
retries: 5
networks:
- app-network
restart: unless-stopped
redis:
image: redis:7-alpine
command: redis-server --appendonly yes
volumes:
- redis-data:/data
networks:
- app-network
restart: unless-stopped
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
depends_on:
- app
networks:
- app-network
restart: unless-stopped
networks:
app-network:
driver: bridge
volumes:
postgres-data:
redis-data:
4. CI/CD Pipeline Setup
Basic Jenkinsfile
// Jenkinsfile
pipeline {
agent any
environment {
DOCKER_IMAGE = "myapp"
DOCKER_TAG = "${env.BUILD_NUMBER}"
DOCKER_REGISTRY = "docker.io/myusername"
}
stages {
stage('Checkout') {
steps {
git branch: 'main',
url: 'https://github.com/username/myapp.git'
}
}
stage('Build') {
steps {
script {
sh 'docker build -t ${DOCKER_IMAGE}:${DOCKER_TAG} .'
sh 'docker tag ${DOCKER_IMAGE}:${DOCKER_TAG} ${DOCKER_IMAGE}:latest'
}
}
}
stage('Test') {
steps {
sh 'docker run --rm ${DOCKER_IMAGE}:${DOCKER_TAG} npm test'
}
}
stage('Push to Registry') {
steps {
script {
docker.withRegistry('https://registry.hub.docker.com', 'docker-hub-credentials') {
sh 'docker push ${DOCKER_REGISTRY}/${DOCKER_IMAGE}:${DOCKER_TAG}'
sh 'docker push ${DOCKER_REGISTRY}/${DOCKER_IMAGE}:latest'
}
}
}
}
stage('Deploy') {
steps {
sh '''
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp -p 3000:3000 ${DOCKER_REGISTRY}/${DOCKER_IMAGE}:${DOCKER_TAG}
'''
}
}
}
post {
success {
echo 'Pipeline succeeded!'
}
failure {
echo 'Pipeline failed!'
}
always {
sh 'docker system prune -f'
}
}
}
Advanced Jenkinsfile with Multiple Environments
// Advanced Jenkinsfile
@Library('shared-library') _
pipeline {
agent {
docker {
image 'maven:3.9-jdk-11'
args '-v $HOME/.m2:/root/.m2'
}
}
parameters {
choice(name: 'ENVIRONMENT', choices: ['dev', 'staging', 'production'], description: 'Deployment environment')
booleanParam(name: 'RUN_TESTS', defaultValue: true, description: 'Run tests?')
booleanParam(name: 'DEPLOY', defaultValue: false, description: 'Deploy to environment?')
}
environment {
APP_NAME = "myapp"
DOCKER_REGISTRY = credentials('docker-registry-url')
DOCKER_CREDENTIALS = credentials('docker-hub')
KUBECONFIG = credentials('kubeconfig')
SONAR_TOKEN = credentials('sonar-token')
}
stages {
stage('Checkout') {
steps {
checkout scm
script {
env.GIT_COMMIT_SHORT = sh(
script: 'git rev-parse --short HEAD',
returnStdout: true
).trim()
}
}
}
stage('Build') {
steps {
sh 'mvn clean package -DskipTests'
archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
}
}
stage('Unit Tests') {
when {
expression { params.RUN_TESTS }
}
steps {
sh 'mvn test'
}
post {
always {
junit 'target/surefire-reports/*.xml'
jacoco execPattern: 'target/jacoco.exec'
}
}
}
stage('Code Quality') {
parallel {
stage('SonarQube') {
steps {
script {
withSonarQubeEnv('SonarQube') {
sh 'mvn sonar:sonar -Dsonar.login=${SONAR_TOKEN}'
}
}
}
}
stage('Dependency Check') {
steps {
sh 'mvn dependency-check:check'
}
}
}
}
stage('Build Docker Image') {
steps {
script {
dockerImage = docker.build("${DOCKER_REGISTRY}/${APP_NAME}:${GIT_COMMIT_SHORT}")
docker.build("${DOCKER_REGISTRY}/${APP_NAME}:latest")
}
}
}
stage('Security Scan') {
steps {
script {
sh "trivy image --severity HIGH,CRITICAL ${DOCKER_REGISTRY}/${APP_NAME}:${GIT_COMMIT_SHORT}"
}
}
}
stage('Push to Registry') {
steps {
script {
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-credentials') {
dockerImage.push()
dockerImage.push('latest')
}
}
}
}
stage('Deploy to Kubernetes') {
when {
expression { params.DEPLOY }
}
steps {
script {
sh """
kubectl set image deployment/${APP_NAME}
${APP_NAME}=${DOCKER_REGISTRY}/${APP_NAME}:${GIT_COMMIT_SHORT}
-n ${params.ENVIRONMENT}
kubectl rollout status deployment/${APP_NAME} -n ${params.ENVIRONMENT}
"""
}
}
}
stage('Integration Tests') {
when {
expression { params.DEPLOY && params.RUN_TESTS }
}
steps {
sh 'mvn verify -Pintegration-tests'
}
}
stage('Smoke Tests') {
when {
expression { params.DEPLOY }
}
steps {
script {
def appUrl = sh(
script: "kubectl get service ${APP_NAME} -n ${params.ENVIRONMENT} -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'",
returnStdout: true
).trim()
sh "curl -f http://${appUrl}/health || exit 1"
}
}
}
}
post {
success {
slackSend(
color: 'good',
message: "Build Success: ${env.JOB_NAME} #${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)"
)
}
failure {
slackSend(
color: 'danger',
message: "Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER} (<${env.BUILD_URL}|Open>)"
)
}
always {
cleanWs()
}
}
}
5. Build and Test Automation
Automated Testing Strategy
# package.json - Node.js example
{
"scripts": {
"test": "jest --coverage",
"test:unit": "jest --testPathPattern=__tests__/unit",
"test:integration": "jest --testPathPattern=__tests__/integration",
"test:e2e": "cypress run",
"lint": "eslint . --ext .js,.jsx,.ts,.tsx",
"lint:fix": "eslint . --ext .js,.jsx,.ts,.tsx --fix",
"security:check": "npm audit",
"security:fix": "npm audit fix"
}
}
# Jenkinsfile test stage
stage('Automated Tests') {
parallel {
stage('Unit Tests') {
steps {
sh 'npm run test:unit'
}
}
stage('Integration Tests') {
steps {
sh 'npm run test:integration'
}
}
stage('E2E Tests') {
steps {
sh 'npm run test:e2e'
}
}
stage('Linting') {
steps {
sh 'npm run lint'
}
}
stage('Security Audit') {
steps {
sh 'npm run security:check'
}
}
}
}
Continuous Testing with Docker
# docker-compose.test.yml
version: '3.8'
services:
test-app:
build:
context: .
dockerfile: Dockerfile.test
environment:
- NODE_ENV=test
- DATABASE_URL=postgresql://postgres:test@test-db:5432/testdb
depends_on:
- test-db
- test-redis
command: npm test
test-db:
image: postgres:15-alpine
environment:
- POSTGRES_DB=testdb
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=test
test-redis:
image: redis:7-alpine
# Run tests in Docker
docker compose -f docker-compose.test.yml up --abort-on-container-exit
6. Deployment Strategies
Blue-Green Deployment
// Jenkinsfile - Blue-Green Deployment
pipeline {
agent any
stages {
stage('Deploy to Green') {
steps {
sh '''
# Deploy new version to green environment
docker stop myapp-green || true
docker rm myapp-green || true
docker run -d --name myapp-green -p 3001:3000 myapp:${BUILD_NUMBER}
# Health check
sleep 10
curl -f http://localhost:3001/health
'''
}
}
stage('Switch Traffic') {
input {
message "Switch traffic to green environment?"
ok "Yes, switch!"
}
steps {
sh '''
# Update load balancer to point to green
# This is environment-specific
# Stop blue environment
docker stop myapp-blue || true
docker rm myapp-blue || true
# Rename green to blue for next deployment
docker rename myapp-green myapp-blue
'''
}
}
}
}
Canary Deployment
# Kubernetes Canary Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-stable
spec:
replicas: 9
selector:
matchLabels:
app: myapp
version: stable
template:
metadata:
labels:
app: myapp
version: stable
spec:
containers:
- name: myapp
image: myapp:v1.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-canary
spec:
replicas: 1 # 10% traffic
selector:
matchLabels:
app: myapp
version: canary
template:
metadata:
labels:
app: myapp
version: canary
spec:
containers:
- name: myapp
image: myapp:v2.0
Rolling Deployment
# Kubernetes Rolling Update
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # Max pods above desired count
maxUnavailable: 1 # Max pods that can be unavailable
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:latest
readinessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 30
periodSeconds: 10
7. Security and Secrets Management
Jenkins Credentials Management
// Using credentials in Jenkinsfile
pipeline {
agent any
stages {
stage('Use Credentials') {
steps {
withCredentials([
usernamePassword(
credentialsId: 'docker-hub',
usernameVariable: 'DOCKER_USER',
passwordVariable: 'DOCKER_PASS'
),
string(
credentialsId: 'api-key',
variable: 'API_KEY'
),
file(
credentialsId: 'kubeconfig',
variable: 'KUBECONFIG_FILE'
)
]) {
sh '''
echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
curl -H "Authorization: Bearer $API_KEY" https://api.example.com
kubectl --kubeconfig=$KUBECONFIG_FILE get pods
'''
}
}
}
}
}
HashiCorp Vault Integration
// Jenkinsfile with Vault
pipeline {
agent any
stages {
stage('Get Secrets from Vault') {
steps {
script {
def secrets = [
[
path: 'secret/data/myapp/db',
engineVersion: 2,
secretValues: [
[envVar: 'DB_USER', vaultKey: 'username'],
[envVar: 'DB_PASS', vaultKey: 'password']
]
]
]
withVault([vaultSecrets: secrets]) {
sh '''
echo "DB User: $DB_USER"
# Use secrets in deployment
'''
}
}
}
}
}
}
Image Security Scanning
# Trivy security scan in Jenkinsfile
stage('Security Scan') {
steps {
script {
// Install Trivy
sh '''
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy
'''
// Scan image
sh '''
trivy image --severity HIGH,CRITICAL --exit-code 1 myapp:latest
'''
}
}
}
# Anchore scan
stage('Anchore Scan') {
steps {
anchore 'myapp:latest'
}
}
8. Monitoring and Observability
Prometheus Metrics in Application
// Node.js application with Prometheus metrics
const express = require('express');
const promClient = require('prom-client');
const app = express();
// Create metrics
const httpRequestDuration = new promClient.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'route', 'status_code']
});
const httpRequestTotal = new promClient.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'route', 'status_code']
});
// Middleware to track metrics
app.use((req, res, next) => {
const start = Date.now();
res.on('finish', () => {
const duration = (Date.now() - start) / 1000;
httpRequestDuration.observe({
method: req.method,
route: req.route?.path || req.path,
status_code: res.statusCode
}, duration);
httpRequestTotal.inc({
method: req.method,
route: req.route?.path || req.path,
status_code: res.statusCode
});
});
next();
});
// Metrics endpoint
app.get('/metrics', async (req, res) => {
res.set('Content-Type', promClient.register.contentType);
res.end(await promClient.register.metrics());
});
app.listen(3000);
Jenkins Pipeline Monitoring
// Jenkinsfile with monitoring
pipeline {
agent any
options {
timestamps()
timeout(time: 1, unit: 'HOURS')
buildDiscarder(logRotator(numToKeepStr: '30'))
}
stages {
stage('Build') {
steps {
script {
def startTime = System.currentTimeMillis()
sh 'docker build -t myapp:latest .'
def duration = System.currentTimeMillis() - startTime
// Send metrics to monitoring system
sh """
curl -X POST http://prometheus-pushgateway:9091/metrics/job/jenkins
--data-binary 'build_duration_seconds{job="myapp",stage="build"} ${duration/1000}'
"""
}
}
}
}
}
9. Best Practices
Pipeline Best Practices
- Version Control Everything: Jenkinsfiles, Dockerfiles, and configuration in Git
- Fast Feedback: Fail fast with quick unit tests before slow integration tests
- Parallel Execution: Run independent stages in parallel
- Idempotent Pipelines: Same input produces same output
- Clean Workspaces: Always clean up after builds
- Artifact Management: Store build artifacts for traceability
- Immutable Artifacts: Never modify artifacts after creation
Docker Best Practices
- Multi-stage Builds: Reduce image size with multi-stage Dockerfiles
- Layer Caching: Order Dockerfile commands for optimal caching
- Security: Run containers as non-root user
- Health Checks: Implement container health checks
- Resource Limits: Set CPU and memory limits
- Image Scanning: Scan images for vulnerabilities
- Tag Properly: Use semantic versioning, not just “latest”
Security Best Practices
- Secrets Management: Never hardcode secrets in code or Dockerfiles
- Least Privilege: Grant minimum required permissions
- Image Scanning: Scan all images before deployment
- Dependency Scanning: Check for vulnerable dependencies
- Network Segmentation: Isolate environments
- Audit Logging: Log all pipeline activities
- Regular Updates: Keep Jenkins and plugins updated
Conclusion
Building robust CI/CD pipelines with Jenkins and Docker enables teams to deliver software faster and more reliably. Key takeaways:
- Automation: Automate build, test, and deployment processes
- Docker: Containerize applications for consistency across environments
- Jenkins: Orchestrate CI/CD workflows with declarative pipelines
- Testing: Implement comprehensive automated testing strategies
- Security: Integrate security scanning and secrets management
- Monitoring: Track pipeline metrics and application health
- Deployment: Use progressive deployment strategies (blue-green, canary)
Start with simple pipelines and gradually add complexity as your team matures in DevOps practices.
Additional Resources
Was this article helpful?
About Ramesh Sundararamaiah
Red Hat Certified Architect
Expert in Linux system administration, DevOps automation, and cloud infrastructure. Specializing in Red Hat Enterprise Linux, CentOS, Ubuntu, Docker, Ansible, and enterprise IT solutions.