**dockerd** - The Docker daemon itself. The highest level component in your list and also the only &#39;Docker&#39; product listed. Provides all the nice UX features of Docker.

**(docker-)containerd** - Also a daemon, listening on a Unix socket, exposes gRPC endpoints. Handles all the low-level container management tasks, storage, image distribution, network attachment, etc...

**(docker-)containerd-ctr** - A lightweight CLI to directly communicate with containerd. Think of it as how &#39;docker&#39; is to &#39;dockerd&#39;.

**(docker-)runc** - A lightweight binary for actually running containers. Deals with the low-level interfacing with Linux capabilities like cgroups, namespaces, etc...

**(docker-)containerd-shim** - After runC actually runs the container, it exits (allowing us to not have any long-running processes responsible for our containers). The shim is the component which sits between containerd and runc to facilitate this.

From the documentation:

&gt; runC is built on libcontainer which is the same container library powering a Docker engine installation. Prior to the version 1.11, Docker engine was used to manage volumes, networks, containers, images etc.. Now, the Docker architecture is broken into four components: 

    Docker engine 
    containerd
    containerd-shm
    runC. 

The binaries are respectively called:

    docker
    docker-containerd
    docker-containerd-shim
    docker-runc.

[![Image][1]][1]

  [1]: https://i.stack.imgur.com/lAtSR.png

https://medium.com/faun/docker-containerd-standalone-runtimes-heres-what-you-should-know-b834ef155426

High-level picture for further better understanding: (read other good answers before this)

[Architecture][1]



**Curated pictures collection for quick understanding:**

[![enter image description here][2]][2]

[![enter image description here][3]][3]


[![enter image description here][4]][4]

Sources: 
1. [Containerd][5]
2. [blog.caoyu.info][6]
3. [Twitter][7]

*There is a reason for this saying &quot;A picture is worth a thousand words&quot;*

  [1]: https://i.stack.imgur.com/SSsmK.png
  [2]: https://i.stack.imgur.com/lWRKb.png
  [3]: https://i.stack.imgur.com/tZwUP.png
  [4]: https://i.stack.imgur.com/KanIf.jpg
  [5]: https://containerd.io/
  [6]: https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fblog.caoyu.info%2Fdocker-world.html&amp;psig=AOvVaw3FCTYHgY0Tei-bn_X6DKYx&amp;ust=1644217125650000&amp;source=images&amp;cd=vfe&amp;ved=0CAsQjRxqFwoTCMjJyJfB6vUCFQAAAAAdAAAAABAK
  [7]: https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Ftwitter.com%2Fiximiuz%2Fstatus%2F1429874962530291717&amp;psig=AOvVaw3FCTYHgY0Tei-bn_X6DKYx&amp;ust=1644217125650000&amp;source=images&amp;cd=vfe&amp;ved=0CAsQjRxqFwoTCMjJyJfB6vUCFQAAAAAdAAAAABAD

I am trying to install Laravel. I have installed `Xampp`, but when I try to setup my database using `php artisan migrate`I get the error:


&gt; **[Illuminate\Database\QueryException]   could not find driver (SQL: select * from information_schema.tables where table_schema = homestead
&gt; and table_name =    migrations)   [PDOException]   could not find
&gt; driver**

`config/database.php` file has the relevant connections:

    &#39;connections&#39; =&gt; [

        &#39;sqlite&#39; =&gt; [
            &#39;driver&#39; =&gt; &#39;sqlite&#39;,
            &#39;database&#39; =&gt; env(&#39;DB_DATABASE&#39;, database_path(&#39;database.sqlite&#39;)),
            &#39;prefix&#39; =&gt; &#39;&#39;,
        ],

        &#39;mysql&#39; =&gt; [
            &#39;driver&#39; =&gt; &#39;mysql&#39;,
            &#39;host&#39; =&gt; env(&#39;DB_HOST&#39;, &#39;127.0.0.1&#39;),
            &#39;port&#39; =&gt; env(&#39;DB_PORT&#39;, &#39;3306&#39;),
            &#39;database&#39; =&gt; env(&#39;DB_DATABASE&#39;, &#39;forge&#39;),
            &#39;username&#39; =&gt; env(&#39;DB_USERNAME&#39;, &#39;forge&#39;),
            &#39;password&#39; =&gt; env(&#39;DB_PASSWORD&#39;, &#39;&#39;),
            &#39;unix_socket&#39; =&gt; env(&#39;DB_SOCKET&#39;, &#39;&#39;),
            &#39;charset&#39; =&gt; &#39;utf8mb4&#39;,
            &#39;collation&#39; =&gt; &#39;utf8mb4_unicode_ci&#39;,
            &#39;prefix&#39; =&gt; &#39;&#39;,
            &#39;strict&#39; =&gt; true,
            &#39;engine&#39; =&gt; null,
        ],

        &#39;pgsql&#39; =&gt; [
            &#39;driver&#39; =&gt; &#39;pgsql&#39;,
            &#39;host&#39; =&gt; env(&#39;DB_HOST&#39;, &#39;127.0.0.1&#39;),
            &#39;port&#39; =&gt; env(&#39;DB_PORT&#39;, &#39;5432&#39;),
            &#39;database&#39; =&gt; env(&#39;DB_DATABASE&#39;, &#39;forge&#39;),
            &#39;username&#39; =&gt; env(&#39;DB_USERNAME&#39;, &#39;forge&#39;),
            &#39;password&#39; =&gt; env(&#39;DB_PASSWORD&#39;, &#39;&#39;),
            &#39;charset&#39; =&gt; &#39;utf8&#39;,
            &#39;prefix&#39; =&gt; &#39;&#39;,
            &#39;schema&#39; =&gt; &#39;public&#39;,
            &#39;sslmode&#39; =&gt; &#39;prefer&#39;,
        ],

        &#39;sqlsrv&#39; =&gt; [
            &#39;driver&#39; =&gt; &#39;sqlsrv&#39;,
            &#39;host&#39; =&gt; env(&#39;DB_HOST&#39;, &#39;localhost&#39;),
            &#39;port&#39; =&gt; env(&#39;DB_PORT&#39;, &#39;1433&#39;),
            &#39;database&#39; =&gt; env(&#39;DB_DATABASE&#39;, &#39;forge&#39;),
            &#39;username&#39; =&gt; env(&#39;DB_USERNAME&#39;, &#39;forge&#39;),
            &#39;password&#39; =&gt; env(&#39;DB_PASSWORD&#39;, &#39;&#39;),
            &#39;charset&#39; =&gt; &#39;utf8&#39;,
            &#39;prefix&#39; =&gt; &#39;&#39;,
        ],

    ],

Any ideas?

Artisan migrate could not find driver

Using [kubernetes-kafka](https://github.com/Yolean/kubernetes-kafka) as a starting point with minikube.

This uses a StatefulSet and a [headless service](https://github.com/Yolean/kubernetes-kafka/blob/master/20dns.yml) for service discovery within the cluster.

The goal is to expose the individual Kafka Brokers externally which are internally addressed as:

    kafka-0.broker.kafka.svc.cluster.local:9092
    kafka-1.broker.kafka.svc.cluster.local:9092 
    kafka-2.broker.kafka.svc.cluster.local:9092

The constraint is that this external service be able to address the brokers specifically.

Whats the right (or one possible) way of going about this? Is it possible to expose a external service per `kafka-x.broker.kafka.svc.cluster.local:9092`?


How to expose a headless service for a StatefulSet externally in Kubernetes

This stuff is really getting confused now. Can someone please explain what&#39;s going on. Just the straight one liner difference.


* `dockerd`
* `libcontainerd`
* `containerd`
* `docker-containerd`
* `docker-runc`
* `docker-containerd-ctr`
* `docker-containerd-shim`

thanks

dockerd vs docker-containerd vs docker-runc vs docker-containerd-ctr vs docker-containerd-shim

<p>This stuff is really getting confused now. Can someone please explain what's going on. Just the straight one liner difference.</p>
<ul>
<li><code>dockerd</code></li>
<li><code>libcontainerd</code></li>
<li><code>containerd</code></li>
<li><code>docker-containerd</code></li>
<li><code>docker-runc</code></li>
<li><code>docker-containerd-ctr</code></li>
<li><code>docker-containerd-shim</code></li>
</ul>
<p>thanks</p>


I got some docker containers running on **AWS EC2**, the /var/lib/docker/overlay2 folder grows very fast in disk size.

I&#39;m wondering if it is safe to delete its content?
or if docker has some kind of command to free up some disk usage.


_________
**UPDATE:**

I actually tried `docker system prune -a` already, which reclaimed 0Kb.

Also my /docker/overlay2 disk size is much larger than the output from `docker system df`

After reading docker documentation and BMitch&#39;s answer, I believe it is a stupid idea to touch this folder and I will try other ways to reclaim my disk space.


Is it safe to clean docker/overlay2/

I&#39;m trying to build a Flask app using Postgres with Docker. I&#39;d like to connect to an AWS RDS instance of Postgres, but use Docker for my Flask app. However, when trying to set up `psycopg2` it runs into an error because it can&#39;t find `pg_config`. Here&#39;s the error:

    Building api
    Step 1/5 : FROM python:3.6.3-alpine3.6
     ---&gt; 84c98ca3b5c5
    Step 2/5 : WORKDIR /usr/src/app
     ---&gt; Using cache
     ---&gt; 407c158f5ee4
    Step 3/5 : COPY . .
     ---&gt; 966df18d329e
    Step 4/5 : RUN pip install -r requirements.txt
     ---&gt; Running in 284cc97aeb63
    Collecting aniso8601==1.3.0 (from -r requirements.txt (line 1))
      Downloading aniso8601-1.3.0.tar.gz (57kB)
    Collecting click==6.7 (from -r requirements.txt (line 2))
      Downloading click-6.7-py2.py3-none-any.whl (71kB)
    Collecting Flask==0.12.2 (from -r requirements.txt (line 3))
      Downloading Flask-0.12.2-py2.py3-none-any.whl (83kB)
    Collecting Flask-RESTful==0.3.6 (from -r requirements.txt (line 4))
      Downloading Flask_RESTful-0.3.6-py2.py3-none-any.whl
    Collecting Flask-SQLAlchemy==2.3.2 (from -r requirements.txt (line 5))
      Downloading Flask_SQLAlchemy-2.3.2-py2.py3-none-any.whl
    Collecting itsdangerous==0.24 (from -r requirements.txt (line 6))
      Downloading itsdangerous-0.24.tar.gz (46kB)
    Collecting Jinja2==2.9.6 (from -r requirements.txt (line 7))
      Downloading Jinja2-2.9.6-py2.py3-none-any.whl (340kB)
    Collecting MarkupSafe==1.0 (from -r requirements.txt (line 8))
      Downloading MarkupSafe-1.0.tar.gz
    Collecting psycopg2==2.7.3.1 (from -r requirements.txt (line 9))
      Downloading psycopg2-2.7.3.1.tar.gz (425kB)
        Complete output from command python setup.py egg_info:
        running egg_info
        creating pip-egg-info/psycopg2.egg-info
        writing pip-egg-info/psycopg2.egg-info/PKG-INFO
        writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
        writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
        writing manifest file &#39;pip-egg-info/psycopg2.egg-info/SOURCES.txt&#39;
        Error: pg_config executable not found.
    
        Please add the directory containing pg_config to the PATH
        or specify the full executable path with the option:
    
            python setup.py build_ext --pg-config /path/to/pg_config build ...
    
        or with the pg_config option in &#39;setup.cfg&#39;.
    
        ----------------------------------------
    Command &quot;python setup.py egg_info&quot; failed with error code 1 in /tmp/pip-build-01lf5grh/psycopg2/
    ERROR: Service &#39;api&#39; failed to build: The command &#39;/bin/sh -c pip install -r requirements.txt&#39; returned a non-zero code: 1

Here&#39;s my `Dockerfile`:

    FROM python:3.6.3-alpine3.6
    
    WORKDIR /usr/src/app
    
    COPY . .
    
    RUN pip install -r requirements.txt
    
    CMD [&quot;python&quot;, &quot;app.py&quot;]

**Many others seem to have a similar issue locally, but none of them involve using Docker. This seems like a Docker issue because I can set up a local virtual environment and the setup works just fine since I have Postgres installed locally and it&#39;s able to find my local `pg_config`.** 

It appears that during the container build/setup, Docker is trying to find `pg_config` within the container. Is there a way to install a `pg_config` in the container, even though I won&#39;t be using a containerized instance of Postgres, but rather the instance on RDS?

Any and all suggestions on how to get around this are welcomed.

Error: pg_config executable not found when installing psycopg2 on Alpine in Docker

There is something that I am not getting when developing an application while using `docker` containers. 

Lets say I am developing a java application and I set up a java container with jdk 8 base image, I still need to install java 8 jdk on my local development machine, since the IDE which I am going to use is going to look for runtime libraries on the local machine not the docker container.

Is this right or am I missing something? Somethings I will be able to do entirely on the docker container like setting up a database but some things I will also have to do on my local development machine and try to match it up with my docker image for example a language runtime like java or python for the sake of using the IDE.

Using an IDE while developing on a docker container

I&#39;m new to ASP.NET Core and docker.
I&#39;ve created a simple ASP.NET Core 2.0 app and try to use docker with it on Windows. However, I get this error: 

`Your Docker server host is configured for &#39;Linux&#39;, however the docker-compose project targets &#39;Windows&#39;.`

Although it seems to be pretty informative error, I can&#39;t find where to &#39;configure host for Windows&#39;

ASP.NET Core docker build error

I&#39;m trying to Dockerize my laravel app.
The app is already built and in git, but I .gitignore my vendor folder.
I&#39;ve added a Dockerfile, which looks like this:
    
    FROM php:7.1-fpm-alpine

    RUN apk update &amp;&amp; apk add curl &amp;&amp; \
      curl -sS https://getcomposer.org/installer | php \
      &amp;&amp; chmod +x composer.phar &amp;&amp; mv composer.phar /usr/local/bin/composer

    RUN apk --no-cache add --virtual .build-deps $PHPIZE_DEPS \
      &amp;&amp; apk --no-cache add --virtual .ext-deps libmcrypt-dev freetype-dev \
      libjpeg-turbo-dev libpng-dev libxml2-dev msmtp bash openssl-dev pkgconfig \
      &amp;&amp; docker-php-source extract \
      &amp;&amp; docker-php-ext-configure gd --with-freetype-dir=/usr/include/ \
                                       --with-png-dir=/usr/include/ \
                                       --with-jpeg-dir=/usr/include/ \
      &amp;&amp; docker-php-ext-install gd mcrypt mysqli pdo pdo_mysql zip opcache \
      &amp;&amp; pecl install mongodb redis xdebug \
      &amp;&amp; docker-php-ext-enable mongodb \
      &amp;&amp; docker-php-ext-enable redis \
      &amp;&amp; docker-php-ext-enable xdebug \
      &amp;&amp; docker-php-source delete \
      &amp;&amp; apk del .build-deps

    WORKDIR /var/www/html

    COPY composer.json composer.lock ./
    RUN composer install --no-scripts --no-autoloader

    COPY . .
    RUN chmod +x artisan

    RUN composer dump-autoload --optimize &amp;&amp; composer run-script post-install-cmd

    CMD php artisan serve --host 0.0.0.0 --port 5001
When I build, this seems to work great. I see the dependencies getting downloaded, I see the autoload file being generated in the output. However, once the build is complete, the vendor folder is not actually there. I&#39;m guessing it was all done in an intermediate container which was then removed? So when I run docker-compose up, I get:
    Fatal error: require(): Failed opening required &#39;/var/www/html/bootstrap/../vendor/autoload.php&#39;

This thread seems to point to the issue - possibly - but doesn&#39;t really provide a solution: https://stackoverflow.com/questions/43769756/composer-install-doesnt-install-packages-when-running-in-dockerfile

Running composer install within a Dockerfile

I am new to dockers. When I am running the docker pull sonarqube I am getting the following error.

**Error response from daemon: Get https://registry-1.docker.io/v2/: dial TCP: lookup registry-1.docker.io on 192.168.65.1:53: no such host**

Can you please let me know why I am getting the error and how can I rectify this.

Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.65.1:53: no such host

Docker Swarm can manage two types of storage:

`volume` and `bind`

While `bind` is not suggested by Docker Documentation since it creates a binding between a local directory (on each swarm Node) to a task, the `volume` implementation is not mentioned, so I don&#39;t understand how volumes are shared between tasks?

- How does Docker Swarm shares volumes between nodes?
- Where are volumes saved (on a manager? And if there is more than one manager?)
- Is there no problem between nodes if it&#39;s running on different machines on different networks?
- Does it create a VPN?

How does Docker Swarm implement volume sharing?

If Docker community runs natively on windows, then why does it need Hyper-v? I.E., doesn&#39;t native imply that Docker-Engine can run instructions on windows? It looks to me like it still starts up a Linux VM and runs with-in that.

To me, is seems that docker-toolbox uses an oracle hyper-visor running linux, while Docker community uses Hyper-V running linux. Is there another important difference that I&#39;m overlooking?

Is this correct? Am I understanding the word &quot;native&quot; wrong, is docker mis-using the word, or is there some other aspect I&#39;m missing?

The reason I&#39;m asking, is because I noticed that you don&#39;t use Docker-machine with the community edition, and I&#39;m wondering why that is. Is docker-machine the thing that runs natiely on windows, while Docker Engine doesn&#39;t? I think the word docker is over-loaded and maybe leads to confusion in this case :)

Thanks in advance!

If Docker runs natively on windows, then why does it need hyper-v

**Issue**: Can not stop docker containers, whenever I try to stop containers I get the following Error message,

    ERROR: for yattyadocker_web_1  cannot stop container: 1f04148910c5bac38983e6beb3f6da4c8be3f46ceeccdc8d7de0da9d2d76edd8: Cannot kill container 1f04148910c5bac38983e6beb3f6da4c8be3f46ceeccdc8d7de0da9d2d76edd8: rpc error: code = PermissionDenied desc = permission denied

**OS Version/build:** Ubuntu 16.04 | Docker Version 17.09.0-ce, build afdb6d4 | Docker Compose version 1.17.1, build 6d101fb

**Steps to reproduce:**

- Created a rails project with Dockerfile and docker-compose.yml. docker-compose.yml is of version 3.
- Image is built successfully with either `docker build -t &lt;project name&gt; . ` or `docker-compose up --build`
- Containers boots up and runs successfully.
- Try to stop docker compose with docker-compose down.

**What I tried:**: 

- I have to run `sudo service docker restart` and then the containers can be removed.
- Uninstalled docker, removed docker directory and then re installed everything. Still facing same issue. 

**Note**: This configuration was working correctly earlier, but somehow file permissions might have changed and I am seeing this error. I have to run `sudo service docker restart` and then the containers can be removed. But this is highly inconvenient and I don&#39;t know how to troubleshoot this.


**Reference Files:**

    # docker-compose.yml
    version: &#39;3&#39;
    volumes:
      db-data:
        driver: local
      redis-data:
        driver: local  
    services:
      db:
        image: postgres:9.4.1
        volumes:
          - db-data:/var/lib/postgresql/data
        ports:
          - &quot;5432:5432&quot;
        env_file: local_envs.env
      web:
        image: yattya_docker:latest
        command: bundle exec puma -C config/puma.rb
        tty: true
        stdin_open: true
        ports:
          - &quot;3000:3000&quot;
        links:
          - db
          - redis
          - memcached
        depends_on:
          - db
          - redis
          - memcached
        env_file: local_envs.env
      redis:
        image: redis:3.2.4-alpine
        ports:
          # We&#39;ll bind our host&#39;s port 6379 to redis&#39;s port 6379, so we can use
          # Redis Desktop Manager (or other tools) with it:
          - 6379:6379
        volumes:
          # We&#39;ll mount the &#39;redis-data&#39; volume into the location redis stores it&#39;s data:
          - redis-data:/var/lib/redis
        command: redis-server --appendonly yes
      memcached:
        image: memcached:1.5-alpine
        ports:
          - &quot;11211:11211&quot;
      clock:
        image: yattya_docker:latest
        command: bundle exec clockwork lib/clock.rb
        links:
          - db
        depends_on:
          - db
        env_file: local_envs.env
      worker:
        image: yattya_docker:latest
        command: bundle exec rake jobs:work
        links: 
          - db
        depends_on: 
          - db
        env_file: local_envs.env

And Dockerfile:

    # Dockerfile
    FROM ruby:2.4.1
    
    RUN apt-get update &amp;&amp; apt-get install -y nodejs --no-install-recommends &amp;&amp; rm -rf /var/lib/apt/lists/*
    
    ENV APP_HOME /app
    RUN mkdir -p $APP_HOME
    WORKDIR $APP_HOME
    
    ADD Gemfile* $APP_HOME/
    RUN bundle install
    
    ADD . $APP_HOME
    
    RUN mkdir -p ${APP_HOME}/log
    RUN cat /dev/null &gt; &quot;$APP_HOME/log/development.log&quot;
    
    RUN mkdir -p ${APP_HOME}/tmp/cache \
        &amp;&amp; mkdir -p ${APP_HOME}/tmp/pids \
        &amp;&amp; mkdir -p ${APP_HOME}/tmp/sockets
    
    EXPOSE 3000



Docker Containers can not be stopped or removed - permission denied Error

In our team, we are currently transitioning to Docker to deploy everything on our server.

We are using Docker Swarm and multiple (10+) compose files defining plenty (20+) of services. Everything works beautifully so far, except when we take down our stack using `docker stack rm &lt;name&gt;` (and redeploy using `docker stack deploy &lt;options&gt; &lt;name&gt;`): about every second time, we get the following error:

    Failed to remove network &lt;id&gt;: Error response from daemon: network &lt;id&gt; not foundFailed to remove some resources from stack: &lt;name&gt;

When using `docker network ls`, the network is indeed not removed, however, `docker network rm &lt;id&gt;` always results in the following:

    Error response from daemon: network &lt;id&gt; not found

What makes that even more strange is the fact that `docker network inspect &lt;id&gt;` returns a normal output. The networks are always `overlay` networks that are created with the compose files used to deploy our stack. Currently, we only have a single node in our Swarm.

Our current &quot;workaround&quot; is to restart Docker (which resolves the issue), but that is not a viable solution in a production environment. Leaving the swarm and joining it again does not resolve the issue either.

At first, we thought that this issue is related to Docker for Mac only (as we first encountered the issue on local machines), however, the same issue arises on Debian Stretch. In both cases, we use the latest Docker distribution available.

I would really appreciate any help!

Content Type	Original Author	Original Content on Stackoverflow
Question	Ijaz Ahmad	View Question on Stackoverflow
Solution 1 - Docker	johnharris85	View Answer on Stackoverflow
Solution 2 - Docker	Ijaz Ahmad	View Answer on Stackoverflow
Solution 3 - Docker	surya	View Answer on Stackoverflow

dockerd vs docker-containerd vs docker-runc vs docker-containerd-ctr vs docker-containerd-shim

Docker Problem Overview

Docker Solutions

Solution 1 - Docker

Solution 2 - Docker

Solution 3 - Docker

How to expose a headless service for a StatefulSet externally in Kubernetes

Artisan migrate could not find driver

Attributions