Código terraform para a pipeline de Deploy
Definição da pipeline
modeldeploy_cd_pipeline.tf
resource "aws_codepipeline" "sm_cd_pipeline" {
name = "modeldeploy-pipeline"
role_arn = aws_iam_role.tf_mlops_role.arn
tags = {
Environment = var.env
}
artifact_store {
location = aws_s3_bucket.artifacts_bucket.bucket
type = "S3"
}
stage {
name = "Source"
action {
category = "Source"
configuration = {
RepositoryName = var.deploy_repository_name
BranchName = var.repository_branch
PollForSourceChanges = true
}
input_artifacts = []
name = "Source"
output_artifacts = [
"SourceArtifact",
]
owner = "AWS"
provider = "CodeCommit"
run_order = 1
version = "1"
}
}
stage {
name = "Build"
action {
category = "Build"
configuration = {
"ProjectName" = "tf-mlops-deploybuild"
}
input_artifacts = [
"SourceArtifact",
]
name = "Build"
output_artifacts = [
"BuildArtifact",
]
owner = "AWS"
provider = "CodeBuild"
run_order = 1
version = "1"
}
}
stage {
name = "DeployStaging"
action {
category = "Deploy"
configuration = {
"ActionMode": "REPLACE_ON_FAILURE",
"Capabilities": "CAPABILITY_NAMED_IAM",
"RoleArn": aws_iam_role.tf_mlops_role.arn,
"StackName": "sagemaker-${var.project_name}-${var.project_id}-deploy-staging",
"TemplateConfiguration": "BuildArtifact::staging-config-export.json",
"TemplatePath": "BuildArtifact::template-export.yml"
}
input_artifacts = [
"BuildArtifact",
]
name = "DeployResourcesStaging"
owner = "AWS"
provider = "CloudFormation"
run_order = 1
version = "1"
}
action {
category = "Build"
configuration = {
"ProjectName" = "tf-mlops-testbuild",
"PrimarySource" = "SourceArtifact"
}
input_artifacts = [
"SourceArtifact","BuildArtifact"
]
name = "TestStaging"
output_artifacts = [
"TestArtifact",
]
owner = "AWS"
provider = "CodeBuild"
run_order = 2
version = "1"
}
action {
category = "Approval"
configuration = {
"CustomData"= "Approve this model for Production"
}
name = "ApproveDeployment"
owner = "AWS"
provider = "Manual"
run_order = 3
version = "1"
}
}
stage {
name = "DeployProd"
action {
category = "Deploy"
configuration = {
"ActionMode": "CREATE_UPDATE",
"RoleArn": aws_iam_role.tf_mlops_role.arn,
"Capabilities": "CAPABILITY_NAMED_IAM",
"StackName": "sagemaker-${var.project_name}-${var.project_id}-deploy-prod",
"TemplateConfiguration": "BuildArtifact::prod-config-export.json",
"TemplatePath": "BuildArtifact::template-export.yml"
}
input_artifacts = [
"BuildArtifact",
]
name = "DeployResourcesProd"
owner = "AWS"
provider = "CloudFormation"
run_order = 1
version = "1"
}
}
}Aqui definimos a pipeline de CD (deploy), a etapa de source fica igual à etapa de Source da pipeline de Build, o código vem do repositório do CodeCommit (repositório de deploy).
Já na etapa de primeira etapa de Build, usamos o tf-mlops-deploybuild, que comentaremos com maior detalhe em seguida.
Em seguida temos a etapa de DeployStaging, que se encarrega de fazer o deploy do endpoint de staging, e rodar testes predefinidos nesse endpoint, esses testes são executados numa segunda etapa de Build, o tf-mlops-testbuild. Depois de feitos o deploy de staging e os testes, a pipeline espera aprovação manual, para prosseguir para o deploy de produção.
O deploy de produção ocorre na etapa de DeployProd, onde usamos o json gerado anteriormente (citado nos templates), assim como o template do CloudFormation para fazer o deploy do Endpoint do SageMaker.
Código da primeira etapa de build.
modeldeploy_buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
python: 3.8
commands:
# Upgrade AWS CLI to the latest version
- pip install --upgrade --force-reinstall botocore boto3 awscli
build:
commands:
# Export the staging and production configuration files
- python build.py --model-execution-role "$MODEL_EXECUTION_ROLE_ARN" --model-package-group-name "$SOURCE_MODEL_PACKAGE_GROUP_NAME" --sagemaker-project-id "$SAGEMAKER_PROJECT_ID" --sagemaker-project-name "$SAGEMAKER_PROJECT_NAME" --export-staging-config $EXPORT_TEMPLATE_STAGING_CONFIG --export-prod-config $EXPORT_TEMPLATE_PROD_CONFIG
# Package the infrastucture as code defined in endpoint-config-template.yml by using AWS CloudFormation.
# Note that the Environment Variables like ARTIFACT_BUCKET, SAGEMAKER_PROJECT_NAME etc,. used below are expected to be setup by the
# CodeBuild resrouce in the infra pipeline (in the ServiceCatalog product)
- aws cloudformation package --template endpoint-config-template.yml --s3-bucket $ARTIFACT_BUCKET --output-template $EXPORT_TEMPLATE_NAME
# Print the files to verify contents
- cat $EXPORT_TEMPLATE_STAGING_CONFIG
- cat $EXPORT_TEMPLATE_PROD_CONFIG
artifacts:
files:
- $EXPORT_TEMPLATE_NAME
- $EXPORT_TEMPLATE_STAGING_CONFIG
- $EXPORT_TEMPLATE_PROD_CONFIG
modeldeploy_codebuild.tf
data "template_file" "deploybuildspec" {
template = file("modeldeploy_buildspec.yml")
vars = {
env = var.env
SAGEMAKER_PROJECT_NAME=var.project_name
SAGEMAKER_PROJECT_ID=var.project_id
ARTIFACT_BUCKET=var.artifacts_bucket_name
MODEL_EXECUTION_ROLE_ARN=aws_iam_role.tf_mlops_role.arn
AWS_REGION=var.region
SOURCE_MODEL_PACKAGE_GROUP_NAME="${var.project_name}-${var.project_id}"
EXPORT_TEMPLATE_NAME="template-export.yml"
EXPORT_TEMPLATE_STAGING_CONFIG="staging-config-export.json"
EXPORT_TEMPLATE_PROD_CONFIG="prod-config-export.json"
}
}
resource "aws_codebuild_project" "tf_mlops_deploybuild" {
badge_enabled = false
build_timeout = 60
name = "tf-mlops-deploybuild"
queued_timeout = 480
service_role = aws_iam_role.tf_mlops_role.arn
tags = {
Environment = var.env
}
artifacts {
encryption_disabled = false
name = "tf-mlops-deploybuild-${var.env}"
override_artifact_name = false
packaging = "NONE"
type = "CODEPIPELINE"
}
environment {
compute_type = "BUILD_GENERAL1_SMALL"
image = "aws/codebuild/amazonlinux2-x86_64-standard:3.0"
image_pull_credentials_type = "CODEBUILD"
privileged_mode = false
type = "LINUX_CONTAINER"
environment_variable {
name = "environment"
type = "PLAINTEXT"
value = var.env
}
environment_variable {
name = "SAGEMAKER_PROJECT_NAME"
type = "PLAINTEXT"
value = var.project_name
}
environment_variable {
name = "SAGEMAKER_PROJECT_ID"
type = "PLAINTEXT"
value = var.project_id
}
environment_variable {
name = "ARTIFACT_BUCKET"
type = "PLAINTEXT"
value = var.artifacts_bucket_name
}
environment_variable {
name = "MODEL_EXECUTION_ROLE_ARN"
type = "PLAINTEXT"
value = aws_iam_role.tf_mlops_role.arn
}
environment_variable {
name = "SOURCE_MODEL_PACKAGE_GROUP_NAME"
type = "PLAINTEXT"
value = "${var.project_name}-${var.project_id}"
}
environment_variable {
name = "AWS_REGION"
type = "PLAINTEXT"
value = var.region
}
environment_variable {
name = "EXPORT_TEMPLATE_NAME"
type = "PLAINTEXT"
value = "template-export.yml"
}
environment_variable {
name = "EXPORT_TEMPLATE_STAGING_CONFIG"
type = "PLAINTEXT"
value = "staging-config-export.json"
}
environment_variable {
name = "EXPORT_TEMPLATE_PROD_CONFIG"
type = "PLAINTEXT"
value = "prod-config-export.json"
}
}
logs_config {
cloudwatch_logs {
status = "ENABLED"
}
s3_logs {
encryption_disabled = false
status = "DISABLED"
}
}
source {
buildspec = data.template_file.deploybuildspec.rendered
git_clone_depth = 0
insecure_ssl = false
report_build_status = false
type = "CODEPIPELINE"
}
}No arquivo yml, é definida os comandos que devem ser executados nessa etapa de build, nela geramos os arquivos de configuração para os deploys, no código terraform todas as variáveis nesse código yml são preenchidas com variáveis do terraform.
O build em si é semelhante ao da pipeline de CI, definimos o nome, a instância, a imagem e as variáveis de ambiente do ambiente de build, assim como definimos o comportamento dos logs, e a fonte do código (temos no buildspec o template yml preenchido com as variáveis).
Código do Build de teste (staging)
modeldeploy_testbuild.tf
resource "aws_codebuild_project" "tf_mlops_testbuild" {
badge_enabled = false
build_timeout = 60
name = "tf-mlops-testbuild"
queued_timeout = 480
service_role = aws_iam_role.tf_mlops_role.arn
tags = {
Environment = var.env
}
artifacts {
encryption_disabled = false
name = "tf-mlops-testbuild-${var.env}"
override_artifact_name = false
packaging = "NONE"
type = "CODEPIPELINE"
}
environment {
compute_type = "BUILD_GENERAL1_SMALL"
image = "aws/codebuild/amazonlinux2-x86_64-standard:2.0"
image_pull_credentials_type = "CODEBUILD"
privileged_mode = false
type = "LINUX_CONTAINER"
environment_variable {
name = "environment"
type = "PLAINTEXT"
value = var.env
}
environment_variable {
name = "SAGEMAKER_PROJECT_NAME"
type = "PLAINTEXT"
value = var.project_name
}
environment_variable {
name = "SAGEMAKER_PROJECT_ID"
type = "PLAINTEXT"
value = var.project_id
}
environment_variable {
name = "AWS_REGION"
type = "PLAINTEXT"
value = var.region
}
environment_variable {
name = "BUILD_CONFIG"
type = "PLAINTEXT"
value = "staging-config-export.json"
}
environment_variable {
name = "EXPORT_TEST_RESULTS"
type = "PLAINTEXT"
value = "test-results.json"
}
}
logs_config {
cloudwatch_logs {
status = "ENABLED"
}
s3_logs {
encryption_disabled = false
status = "DISABLED"
}
}
source {
buildspec = "test/test_buildspec.yml"
git_clone_depth = 0
insecure_ssl = false
report_build_status = false
type = "CODEPIPELINE"
}
}Como dito anteriormente, essa etapa tem como objetivo rodar testes no endpoint de staging, para garantir que tudo está correto antes de botar em produção.
O código em si é bem semelhante aos anteriores, mudando apenas algumas variáveis de ambiente, como BUILD_CONFIG e EXPORT_TEST_RESULTS e o buildspec de source, que usa um yml contido nos templates.