bitbucket - Clone private git repo with dockerfile

ID : 10445

viewed : 33

Tags : gitbitbucketdockergit

Top 5 Answer for bitbucket - Clone private git repo with dockerfile

vote vote

90

My key was password protected which was causing the problem, a working file is now listed below (for help of future googlers)

FROM ubuntu  MAINTAINER Luke Crooks "luke@pumalo.org"  # Update aptitude with new repo RUN apt-get update  # Install software  RUN apt-get install -y git # Make ssh dir RUN mkdir /root/.ssh/  # Copy over private key, and set permissions # Warning! Anyone who gets their hands on this image will be able # to retrieve this private key file from the corresponding image layer ADD id_rsa /root/.ssh/id_rsa  # Create known_hosts RUN touch /root/.ssh/known_hosts # Add bitbuckets key RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts  # Clone the conf files into the docker container RUN git clone git@bitbucket.org:User/repo.git 
vote vote

82

You should create new SSH key set for that Docker image, as you probably don't want to embed there your own private key. To make it work, you'll have to add that key to deployment keys in your git repository. Here's complete recipe:

  1. Generate ssh keys with ssh-keygen -q -t rsa -N '' -f repo-key which will give you repo-key and repo-key.pub files.

  2. Add repo-key.pub to your repository deployment keys.
    On GitHub, go to [your repository] -> Settings -> Deploy keys

  3. Add something like this to your Dockerfile:

     ADD repo-key / RUN \   chmod 600 /repo-key && \     echo "IdentityFile /repo-key" >> /etc/ssh/ssh_config && \     echo -e "StrictHostKeyChecking no" >> /etc/ssh/ssh_config && \     // your git clone commands here... 

Note that above switches off StrictHostKeyChecking, so you don't need .ssh/known_hosts. Although I probably like more the solution with ssh-keyscan in one of the answers above.

vote vote

70

There's no need to fiddle around with ssh configurations. Use a configuration file (not a Dockerfile) that contains environment variables, and have a shell script update your docker file at runtime. You keep tokens out of your Dockerfiles and you can clone over https (no need to generate or pass around ssh keys).

Go to Settings > Personal Access Tokens

  • Generate a personal access token with repo scope enabled.
  • Clone like this: git clone https://MY_TOKEN@github.com/user-or-org/repo

Some commenters have noted that if you use a shared Dockerfile, this could expose your access key to other people on your project. While this may or may not be a concern for your specific use case, here are some ways you can deal with that:

  • Use a shell script to accept arguments which could contain your key as a variable. Replace a variable in your Dockerfile with sed or similar, i.e. calling the script with sh rundocker.sh MYTOKEN=foo which would replace on https://{{MY_TOKEN}}@github.com/user-or-org/repo. Note that you could also use a configuration file (in .yml or whatever format you want) to do the same thing but with environment variables.
  • Create a github user (and generate an access token for) for that project only
vote vote

68

You often do not want to perform a git clone of a private repo from within the docker build. Doing the clone there involves placing the private ssh credentials inside the image where they can be later extracted by anyone with access to your image.

Instead, the common practice is to clone the git repo from outside of docker in your CI tool of choice, and simply COPY the files into the image. This has a second benefit: docker caching. Docker caching looks at the command being run, environment variables it includes, input files, etc, and if they are identical to a previous build from the same parent step, it reuses that previous cache. With a git clone command, the command itself is identical, so docker will reuse the cache even if the external git repo is changed. However, a COPY command will look at the files in the build context and can see if they are identical or have been updated, and use the cache only when it's appropriate.


If you are going to add credentials into your build, consider doing so with a multi-stage build, and only placing those credentials in an early stage that is never tagged and pushed outside of your build host. The result looks like:

FROM ubuntu as clone  # Update aptitude with new repo RUN apt-get update \  && apt-get install -y git # Make ssh dir # Create known_hosts # Add bitbuckets key RUN mkdir /root/.ssh/ \  && touch /root/.ssh/known_hosts \  && ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts  # Copy over private key, and set permissions # Warning! Anyone who gets their hands on this image will be able # to retrieve this private key file from the corresponding image layer COPY id_rsa /root/.ssh/id_rsa  # Clone the conf files into the docker container RUN git clone git@bitbucket.org:User/repo.git  FROM ubuntu as release LABEL maintainer="Luke Crooks <luke@pumalo.org>"  COPY --from=clone /repo /repo ... 

More recently, BuildKit has been testing some experimental features that allow you to pass an ssh key in as a mount that never gets written to the image:

# syntax=docker/dockerfile:experimental FROM ubuntu as clone LABEL maintainer="Luke Crooks <luke@pumalo.org>"  # Update aptitude with new repo RUN apt-get update \  && apt-get install -y git  # Make ssh dir # Create known_hosts # Add bitbuckets key RUN mkdir /root/.ssh/ \  && touch /root/.ssh/known_hosts \  && ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts  # Clone the conf files into the docker container RUN --mount=type=secret,id=ssh_id,target=/root/.ssh/id_rsa \     git clone git@bitbucket.org:User/repo.git 

And you can build that with:

$ DOCKER_BUILDKIT=1 docker build -t your_image_name \   --secret id=ssh_id,src=$(pwd)/id_rsa . 

Note that this still requires your ssh key to not be password protected, but you can at least run the build in a single stage, removing a COPY command, and avoiding the ssh credential from ever being part of an image.


BuildKit also added a feature just for ssh which allows you to still have your password protected ssh keys, the result looks like:

# syntax=docker/dockerfile:experimental FROM ubuntu as clone LABEL maintainer="Luke Crooks <luke@pumalo.org>"  # Update aptitude with new repo RUN apt-get update \  && apt-get install -y git  # Make ssh dir # Create known_hosts # Add bitbuckets key RUN mkdir /root/.ssh/ \  && touch /root/.ssh/known_hosts \  && ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts  # Clone the conf files into the docker container RUN --mount=type=ssh \     git clone git@bitbucket.org:User/repo.git 

And you can build that with:

$ eval $(ssh-agent) $ ssh-add ~/.ssh/id_rsa (Input your passphrase here) $ DOCKER_BUILDKIT=1 docker build -t your_image_name \   --ssh default=$SSH_AUTH_SOCK . 

Again, this is injected into the build without ever being written to an image layer, removing the risk that the credential could accidentally leak out.


To force docker to run the git clone even when the lines before have been cached, you can inject a build ARG that changes with each build to break the cache. That looks like:

# inject a datestamp arg which is treated as an environment variable and # will break the cache for the next RUN command ARG DATE_STAMP # Clone the conf files into the docker container RUN git clone git@bitbucket.org:User/repo.git 

Then you inject that changing arg in the docker build command:

date_stamp=$(date +%Y%m%d-%H%M%S) docker build --build-arg DATE_STAMP=$date_stamp . 
vote vote

55

Another option is to use a multi-stage docker build to ensure that your SSH keys are not included in the final image.

As described in my post you can prepare your intermediate image with the required dependencies to git clone and then COPY the required files into your final image.

Additionally if we LABEL our intermediate layers, we can even delete them from the machine when finished.

# Choose and name our temporary image. FROM alpine as intermediate # Add metadata identifying these images as our build containers (this will be useful later!) LABEL stage=intermediate  # Take an SSH key as a build argument. ARG SSH_KEY  # Install dependencies required to git clone. RUN apk update && \     apk add --update git && \     apk add --update openssh  # 1. Create the SSH directory. # 2. Populate the private key file. # 3. Set the required permissions. # 4. Add github to our list of known hosts for ssh. RUN mkdir -p /root/.ssh/ && \     echo "$SSH_KEY" > /root/.ssh/id_rsa && \     chmod -R 600 /root/.ssh/ && \     ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts  # Clone a repository (my website in this case) RUN git clone git@github.com:janakerman/janakerman.git  # Choose the base image for our final image FROM alpine  # Copy across the files from our `intermediate` container RUN mkdir files COPY --from=intermediate /janakerman/README.md /files/README.md 

We can then build:

MY_KEY=$(cat ~/.ssh/id_rsa) docker build --build-arg SSH_KEY="$MY_KEY" --tag clone-example . 

Prove our SSH keys are gone:

docker run -ti --rm clone-example cat /root/.ssh/id_rsa 

Clean intermediate images from the build machine:

docker rmi -f $(docker images -q --filter label=stage=intermediate) 

Top 3 video Explaining bitbucket - Clone private git repo with dockerfile

Related QUESTION?