Issue

This Content is from Stack Overflow. Question asked by Diego Cândido da Silva

I have a pipeline where I’m trying to reuse the maven cache downloaded in the first job (.m2) in other jobs, but it seems that the azure pipeline deletes the files before each job. Is there any way to keep a folder and its files between jobs ?

my azure-pipeline.yml file:

when executing the tree command in the second job in the $(MAVEN_CACHE_FOLDER) directory, azure returns the following message: /home/vsts/work/1/a/.m2/repository [error opening dir]

pool:
  vmImage: ubuntu-latest

variables:
  MAVEN_CACHE_FOLDER: $(Build.ArtifactStagingDirectory)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

stages:
  - stage: validate
    displayName: VALIDATE
    jobs:
      - job: java_validations
        displayName: Java Validations
        workspace:
          clean: outputs
        steps:
          - checkout: self
            clean: false
          - task: Cache@2
            displayName: Cache Maven local repo
            inputs:
              key: 'maven | $(Agent.OS) | **/pom.xml, !**/target/**'
              restoreKeys: |
                maven | $(Agent.OS)
                maven
              path: $(MAVEN_CACHE_FOLDER)
          - script: |
              mvn validate $(MAVEN_OPTS)
            displayName: Maven Validate
          - script: |
              mvn checkstyle:check $(MAVEN_OPTS)
            displayName: Maven Checkstyle
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
      - job: unit_tests
        displayName: Unit Tests
        dependsOn:
          - java_validations
        workspace:
          clean: outputs
        steps:
          - checkout: none
            clean: false
          - script: |
              tree $(MAVEN_CACHE_FOLDER)
            displayName: show MAVEN_CACHE_FOLDER tree
          - task: Maven@3
            inputs:
              mavenPomFile: 'pom.xml'
              options: 'test-compile failsafe:integration-test -Dcheckstyle.skip -Pun-tests $(MAVEN_OPTS)'
              publishJUnitResults: false
              mavenVersionOption: 'Default'
              mavenAuthenticateFeed: true
              effectivePomSkip: false
              sonarQubeRunAnalysis: true
              sqMavenPluginVersionChoice: 'latest'
            env:
              JAVA_HOME: $(JAVA_HOME_17_X64)
              PATH: $(JAVA_HOME_17_X64)/bin:$(PATH)

Solution

Refer to this doc: Microsoft-hosted agents

Each time you run a pipeline, you get a fresh virtual machine for each job in the pipeline. The virtual machine is discarded after one job (which means any change that a job makes to the virtual machine file system, such as checking out code, will be unavailable to the next job).

From your YAML sample, you are using Microsoft-Hosted ubuntu agent. The two jobs are running on different virtual machines. So the files will not keep between jobs.

Here are the workarounds:

You can create a Self-hosted agent. Then two jobs will run on the same agent and the files can be used between jobs.
You can publish the files to Build artifacts in Job 1, and then you can download the files to the agent in Job 2. Then the files in Job 1 can be used in Job 2.

For more detailed info, you can refer to this doc: Publish and download pipeline Artifacts

This Question was asked in StackOverflow by Diego Cândido da Silva and Answered by Kevin Lu-MSFT It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.