Module 8 : Bonnes Pratiques DevOps

Templates de pipeline

Les templates permettent de réutiliser du code pipeline :

# templates/build-python.yml
parameters:
  - name: pythonVersion
    default: '3.10'
  - name: workingDirectory
    default: '.'

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '${{ parameters.pythonVersion }}'

  - script: |
      cd ${{ parameters.workingDirectory }}
      pip install -r requirements.txt
    displayName: 'Install dependencies'

  - script: |
      cd ${{ parameters.workingDirectory }}
      pytest tests/ --junitxml=test-results.xml
    displayName: 'Run tests'

# Utilisation
# azure-pipelines.yml
stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - template: templates/build-python.yml
            parameters:
              pythonVersion: '3.10'
              workingDirectory: 'src'

Templates de stages

# templates/deploy-synapse.yml
parameters:
  - name: environment
  - name: resourceGroup
  - name: synapseWorkspace

stages:
  - stage: Deploy_${{ parameters.environment }}
    jobs:
      - deployment: DeploySynapse
        environment: ${{ parameters.environment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: 'azure-${{ parameters.environment }}'
                    scriptType: 'pscore'
                    scriptLocation: 'inlineScript'
                    inlineScript: |
                      az synapse pipeline create --workspace-name ${{ parameters.synapseWorkspace }} --name "$(Pipeline.Name)" --file @pipelines/definition.json

# Utilisation
stages:
  - template: templates/deploy-synapse.yml
    parameters:
      environment: 'dev'
      resourceGroup: 'rg-folab-dev'
      synapseWorkspace: 'syn-folab-dev'

  - template: templates/deploy-synapse.yml
    parameters:
      environment: 'prod'
      resourceGroup: 'rg-folab-prod'
      synapseWorkspace: 'syn-folab-prod'

Organisation du code

# Structure recommandée pour un projet Synapse
synapse-folab/
|
+-- .azure-pipelines/
|   +-- ci.yml                 # Pipeline CI
|   +-- cd.yml                 # Pipeline CD
|   +-- templates/             # Templates réutilisables
|
+-- src/
|   +-- pipelines/             # Definitions Synapse pipelines
|   +-- notebooks/             # Notebooks PySpark
|   +-- sql/                   # Scripts SQL
|   +-- linkedServices/        # Linked Services
|   +-- datasets/              # Datasets
|
+-- tests/
|   +-- unit/                  # Tests unitaires
|   +-- integration/           # Tests integration
|
+-- scripts/
|   +-- deploy.sh              # Scripts de déploiement
|   +-- validate.py            # Scripts de validation
|
+-- docs/
|   +-- architecture.md
|   +-- runbook.md
|
+-- README.md
+-- requirements.txt

Infrastructure as Code

# Bicep pour l'infrastructure Azure
# infra/main.bicep
param environment string
param location string = 'westeurope'

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: 'st${environment}folab'
  location: location
  sku: { name: 'Standard_LRS' }
  kind: 'StorageV2'
  properties: {
    isHnsEnabled: true
  }
}

resource synapseWorkspace 'Microsoft.Synapse/workspaces@2021-06-01' = {
  name: 'syn-${environment}-folab'
  location: location
  properties: {
    defaultDataLakeStorage: {
      accountUrl: storageAccount.properties.primaryEndpoints.dfs
      filesystem: 'datalake'
    }
  }
}

# Pipeline pour deployer l'infra
- stage: DeployInfra
  jobs:
    - job: DeployBicep
      steps:
        - task: AzureCLI@2
          inputs:
            azureSubscription: 'azure-dev'
            scriptType: 'pscore'
            scriptLocation: 'inlineScript'
            inlineScript: |
              az deployment group create --resource-group rg-folab-dev --template-file infra/main.bicep --parameters environment=dev

# Pipeline pour deployer l'infra (Bash)
- stage: DeployInfra
  jobs:
    - job: DeployBicep
      steps:
        - task: AzureCLI@2
          inputs:
            azureSubscription: 'azure-dev'
            scriptType: 'bash'
            scriptLocation: 'inlineScript'
            inlineScript: |
              az deployment group create \
                --resource-group rg-folab-dev \
                --template-file infra/main.bicep \
                --parameters environment=dev

Monitoring des pipelines

# Métriques à suivre
- Build success rate (> 95%)
- Build duration (< 10 min pour CI)
- Deployment frequency (plusieurs fois par jour)
- Lead time (code commit -> production)
- Mean time to recovery (MTTR)

# Alertes recommandées
- Build failure sur main
- Déploiement Prod échoué
- Build duration > seuil
- Tests coverage < seuil

Documentation

# README.md exemple
# FOLAB Synapse Pipelines

## Prerequisites
- Azure CLI
- Python 3.10+
- Access to Azure DevOps

## Local Development
```bash
pip install -r requirements.txt
pytest tests/
```

## Deployment
Deployments are automated via Azure Pipelines.
- Push to `develop` -> Deploy to Dev
- Merge to `main` -> Deploy to Prod (with approval)

## Architecture
See [docs/architecture.md](docs/architecture.md)

Checklist projet DevOps