how to disable direct access to a web site by ip address

I have a website on a VPS.

The issue I am having is that when I enter the IP of the server, it links to the website.
Even when entering mail.domain.com, it does the same thing.

How do I disable that, so a visitor would get a message or be directed to the domain?

I tried disabling the IP and mail a record on cloud flare but it didn't work.

My setup is:

VPS on Linux Debian 
Nginx
no control panel just command line
Cloudflare
DNS setup with BIND

server {
    listen      80 default_server;
    listen      [::]:80 default_server;
    server_name "";
    return      444;
}

You need to specify default_server parameter so that all non available server requests goes to this server block which throws 444 error.

444 : CONNECTION CLOSED WITHOUT RESPONSE

ref: https://httpstatuses.com/444

You can use redirect, nginx config:

server {
        listen 80;
        server_name IP_ADDRESS;
        return 301 http://YOUR.DOMAIN;
}

You can just add a server directive before others.

server {
    listen 80;
    server_name _;
    return 404;
}

You can use redirect, nginx config:

server {
        listen 80;`enter code here`
        server_name IP_ADDRESS;
        return 301 http://YOUR.DOMAIN;
}

you can return any error you find suitable. A list of errors can be found here List_of_HTTP_status_codes

server {
    listen      x.x.x.x:80;
    server_name x.x.x.x;
    return      404;
}  

You may try to set the server IP address in:

/etc/nginx/conf.d/default.conf

So it looks like this:

server {
    listen 80;
    server_name localhost IP.OF.VPS.HERE;

Then you can specify the subdomain vhost, like:

server {
        listen 80;
        server_name subdomain.domain.com;

And the main domain, like:

server {
        listen 80;
        server_name www.domain.com domain.com;

Then restart Nginx:

/etc/init.d/nginx restart

Each vhost should have its own *.conf file (for better organization), like:

/etc/nginx/conf.d/subdomain.domain.com.conf
/etc/nginx/conf.d/domain.com.conf
/etc/nginx/conf.d/default.conf

Put this at top of your /etc/nginx/conf.d/SERVER_IP_ADDRESS.conf file and comment everything what is below it.

#disabling accesing server by ip address
server {
        listen SERVER_IP_ADDRESS:80 default;
        server_name _;
        return 404;
}

Then restart your Nginx server (on Ubuntu it is done by service nginx restart this command)

Now when you will put your server's ip address to browser url field you will get 404 error.

if ($http_host != "example.com") {
    return 301 https://example.com;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    listen 443 default_server;
    listen [::]:443 default_server;
    return 444;
}

Don't bother supporting HTTP/2 or SSL connection for a sink. It does catch-all without those unnecesities. For the unsupported, it just refuses such connections.

See https://stackoverflow.com/a/68042877/4510033.