What is d_type and why Docker overlayfs need it
2017-09-14 22:22
4083 查看
https://linuxer.pro/2017/03/what-is-d_type-and-why-docker-overlayfs-need-it/
In my previous post I’ve mentioned a strange problem that occurs on Discourse running in Docker. Today I’m going to explain this further
as this problem could potentially impact any Docker setup uses overlayfs storage driver. Practically, CentOS 7 with all default setup during installation is 100% affected. Docker on Ubuntu uses AUFS so is not affected.
d_type is the term used in Linux kernel that stands for “directory entry type”. Directory entry is a data structure that Linux kernel used to describe the information of a directory on the filesystem. d_type is a field in that data structure which represents
the type of an “file” the directory entry points to. It could a directory, a regular file, some special file such as a pipe, a char device, a socket file etc.
d_type information was added to Linux kernel version 2.6.4. Since then Linux filesystems started to implement it over time. However still some filesystem don’t implement yet, some implement it in a optional way, i.e. it could be enabled or disabled depends
on how the user creates the filesystem.
Overlay and Overlay2 are the two supported storage driver of Docker. Both of them depends on the overlayfs filesystem. Below is a picture from Docker document shows how Docker uses overlayfs for its image storage.
In the overlayfs code(it’s part of the Linux kernel), this d_type information is accessed and used to make sure some file operations are correctly handled. There is code in overlayfs to specifically check for existence of the d_type feature, and print warning
message if it does not exist on the underlying filesystem.
Docker, when running on overlay/overlay2 storage driver, requires the d_type feature to functioning correctly. A check was added to Docker 1.13. By running
tell whether your backing filesystems supports it or not. The plan is to issue an error message in Docker 1.16 if d_type is not enabled.
When d_type is no supported on the backing filesystem of overlayfs, containers running on Docker would run into some strange errors doing file operation. Chown error during Discourse bootstrap or rebuild is one common error. There are some other examples you
can find in Docker issues on GitHub, I’ve take some for example as below.
Randomly cannot start Containers with “Clean up Error! Cannot destroy container” “mkdir …-init/merged/dev/shm: invalid argument” #22937
Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2. #27358
docker run fails with “invalid argument” when using overlay driver on top of xfs #10294
TL;DR: Ext4? Good. XFS on RHEL/CentOS 7? High chance bad, use
As mentioned above, d_type support is optional for some filesystem. This includes XFS, the default filesystem in Red Hat Enterprise Linux 7, which is the upstream base of CentOS 7. Unfortunately, the Red Hat /CentOS installer and mkfs.xfs command all by default
create XFS filesystem without d_type feature turned on…… What a mess!
As a quick rule, if you are using RHEL 7 or CentOS 7, and your filesystem is created by default without specifying an parameter, you can almost be 100% sure that d_type is not turned on on your filesystem. To check for sure, follow below steps.
First you need to find out what filesystem you are currently using. Although XFS is the default during installation, some people or the hosting provider may choose to use Ext4. If that’s the case, then relax, d_type is supported.
If you are on XFS, then you need to run
Pay attention to the last column of 6th line of the command output. You can see ftype=1. That’s a good news. It means my XFS was created with the correct parameter, ftype=1, thus d_type was turned on. If you see a ftype=0 there, that means d_type is off.
Another bad news is this problem can only be fixed by recreate the filesystem. It cannot be change on an existing filesystem! Basically the steps are:
Backup your data
Recreate the filesystem with correct parameter for XFS, or just create an Ext4 filesystem instead.
Restore your data back.
Let’s focus on step #2. DON’T try any of below command on your server before you fully understand them and have backup secured!
If you chose Ext4 filesystem, then it’s easy, just run
If you chose XFS filesystem, the correct command is:
The
It is a good idea to check your system asap to see if this d_type problem affects your RHEL/CentOS 7 installation. The sooner you fix the problem the better.
In my previous post I’ve mentioned a strange problem that occurs on Discourse running in Docker. Today I’m going to explain this further
as this problem could potentially impact any Docker setup uses overlayfs storage driver. Practically, CentOS 7 with all default setup during installation is 100% affected. Docker on Ubuntu uses AUFS so is not affected.
What is d_type
d_type is the term used in Linux kernel that stands for “directory entry type”. Directory entry is a data structure that Linux kernel used to describe the information of a directory on the filesystem. d_type is a field in that data structure which representsthe type of an “file” the directory entry points to. It could a directory, a regular file, some special file such as a pipe, a char device, a socket file etc.
d_type information was added to Linux kernel version 2.6.4. Since then Linux filesystems started to implement it over time. However still some filesystem don’t implement yet, some implement it in a optional way, i.e. it could be enabled or disabled depends
on how the user creates the filesystem.
Why it is important to Docker
Overlay and Overlay2 are the two supported storage driver of Docker. Both of them depends on the overlayfs filesystem. Below is a picture from Docker document shows how Docker uses overlayfs for its image storage.In the overlayfs code(it’s part of the Linux kernel), this d_type information is accessed and used to make sure some file operations are correctly handled. There is code in overlayfs to specifically check for existence of the d_type feature, and print warning
message if it does not exist on the underlying filesystem.
Docker, when running on overlay/overlay2 storage driver, requires the d_type feature to functioning correctly. A check was added to Docker 1.13. By running
docker infocommand now you can
tell whether your backing filesystems supports it or not. The plan is to issue an error message in Docker 1.16 if d_type is not enabled.
When d_type is no supported on the backing filesystem of overlayfs, containers running on Docker would run into some strange errors doing file operation. Chown error during Discourse bootstrap or rebuild is one common error. There are some other examples you
can find in Docker issues on GitHub, I’ve take some for example as below.
Randomly cannot start Containers with “Clean up Error! Cannot destroy container” “mkdir …-init/merged/dev/shm: invalid argument” #22937
Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2. #27358
docker run fails with “invalid argument” when using overlay driver on top of xfs #10294
Check whether your system is affected
TL;DR: Ext4? Good. XFS on RHEL/CentOS 7? High chance bad, use xfs_infoto confirm
As mentioned above, d_type support is optional for some filesystem. This includes XFS, the default filesystem in Red Hat Enterprise Linux 7, which is the upstream base of CentOS 7. Unfortunately, the Red Hat /CentOS installer and mkfs.xfs command all by default
create XFS filesystem without d_type feature turned on…… What a mess!
As a quick rule, if you are using RHEL 7 or CentOS 7, and your filesystem is created by default without specifying an parameter, you can almost be 100% sure that d_type is not turned on on your filesystem. To check for sure, follow below steps.
First you need to find out what filesystem you are currently using. Although XFS is the default during installation, some people or the hosting provider may choose to use Ext4. If that’s the case, then relax, d_type is supported.
If you are on XFS, then you need to run
xfs_infocommand against the filesystem you need to check. Below is an example from my system.
$ xfs_info / meta-data=/dev/sda1 isize=256 agcount=4, agsize=3276736 blks = sectsz=512 attr=2, projid32bit=1 = crc=0 finobt=0 spinodes=0 data = bsize=4096 blocks=13106944, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=6399, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
Pay attention to the last column of 6th line of the command output. You can see ftype=1. That’s a good news. It means my XFS was created with the correct parameter, ftype=1, thus d_type was turned on. If you see a ftype=0 there, that means d_type is off.
How to solve the problem
Another bad news is this problem can only be fixed by recreate the filesystem. It cannot be change on an existing filesystem! Basically the steps are:Backup your data
Recreate the filesystem with correct parameter for XFS, or just create an Ext4 filesystem instead.
Restore your data back.
Let’s focus on step #2. DON’T try any of below command on your server before you fully understand them and have backup secured!
If you chose Ext4 filesystem, then it’s easy, just run
mkfs.ext4 /path/to/your/deviceand that’s it.
If you chose XFS filesystem, the correct command is:
mkfs.xfs -n ftype=1 /path/to/your/device
The
-n ftype=1parameter tells
mkfs.xfsprogram to create a XFS with d_type feature turned on.
Take actions
It is a good idea to check your system asap to see if this d_type problem affects your RHEL/CentOS 7 installation. The sooner you fix the problem the better.
相关文章推荐
- Docker Orchestration... What It Means and Why You Need It (Docker 编配 ...它是什么意思,为什么你会需要它?)
- What is Docker and why is it so darn popular?
- Requirement Traceability Matrix (RTM) – What Is RTM And Why Do We Need It?
- what is the virtual machine, when and why we need use it ?
- NSLock(Mac OS X (Cocoa) ) — Simple Explanation of What It Is and Why You Need It
- What is svchost.exe And Why Is It Running?
- add cool toolbar 256corler is easy,only two file and onCreate func is needed also the MainFrame.toolbar must resize to what real need and also the load width is same to it.
- What is the GlobalSuppressions.cs/GlobalSuppressions.vb file and why is it needed
- What is Event Driven Architecture (EDA) and Why Does it Matter EDA 摘要
- What is virtual memory, how is it implemented, and why do operating systems use it?
- What is initrd and why is it necessary
- 《What is deep learning and why is it getting so much attention?》,译名:深度学习的定义及其备受关注的原因
- VS 2005 Web Project System: What is it and why did we do it?
- What Is UTF-8 And Why Is It Important?
- 【html】What is href=“#” and why is it used?
- What is a magic number? And why is it bad?
- (Page 1 of 3 )A walking tour of JavaBeans What JavaBeans is, how it works, and why you want to use it
- node-haystack Episode 1: What is it and why
- (Page 2 of 3 )A walking tour of JavaBeans 2 :What JavaBeans is, how it works, and why you want to use it