What is the smartest way to handle robots.txt in Express?
node.jsExpressrobots.txtnode.js Problem Overview
I'm currently working on an application built with Express (Node.js) and I want to know what is the smartest way to handle different robots.txt for different environments (development, production).
This is what I have right now but I'm not convinced by the solution, I think it is dirty:
app.get '/robots.txt', (req, res) ->
res.set 'Content-Type', 'text/plain'
if app.settings.env == 'production'
res.send 'User-agent: *\nDisallow: /signin\nDisallow: /signup\nDisallow: /signout\nSitemap: /sitemap.xml'
else
res.send 'User-agent: *\nDisallow: /'
(NB: it is CoffeeScript)
There should be a better way. How would you do it?
Thank you.
node.js Solutions
Solution 1 - node.js
Use a middleware function. This way the robots.txt will be handled before any session, cookieParser, etc:
app.use('/robots.txt', function (req, res, next) {
res.type('text/plain')
res.send("User-agent: *\nDisallow: /");
});
With express 4 app.get now gets handled in the order it appears so you can just use that:
app.get('/robots.txt', function (req, res) {
res.type('text/plain');
res.send("User-agent: *\nDisallow: /");
});
Solution 2 - node.js
1. Create robots.txt with following content :
User-agent: *
Disallow: # your rules here
2. Add it to public/ directory.
3. If not already present in your code, add:
app.use(express.static('public'))
Your robots.txt will be available to any crawler at http://yoursite.com/robots.txt
Solution 3 - node.js
Looks like an ok way.
An alternative, if you'd like to be able to edit robots.txt as regular file, and possibly have other files you only want in production or development mode would be to use 2 separate directories, and activate one or the other at startup.
if (app.settings.env === 'production') {
app.use(express['static'](__dirname + '/production'));
} else {
app.use(express['static'](__dirname + '/development'));
}
then you add 2 directories with each version of robots.txt.
PROJECT DIR
development
robots.txt <-- dev version
production
robots.txt <-- more permissive prod version
And you can keep adding more files in either directory and keep your code simpler.
(sorry, this is javascript, not coffeescript)
Solution 4 - node.js
Here is what I use
router.use('/robots.txt', function (req, res, next) {
res.type('text/plain')
res.send(
`User-agent: *
Disallow: /admin`);
});
Solution 5 - node.js
For choosing the robots.txt depending the environment with a middleware way:
var env = process.env.NODE_ENV || 'development';
if (env === 'development' || env === 'qa') {
app.use(function (req, res, next) {
if ('/robots.txt' === req.url) {
res.type('text/plain');
res.send('User-agent: *\nDisallow: /');
} else {
next();
}
});
}
Solution 6 - node.js
This is what I did on my index routes. You can just simply write down in your codes what I does given down below.
router.get('/', (req, res) =>
res.sendFile(__dirname + '/public/sitemap.xml')
)
router.get('/', (req, res) => {
res.sendFile(__dirname + '/public/robots.txt')
})
Solution 7 - node.js
I use robots.txt as a normal file for Prod, and a middleware for other envs.
if(isDev || isStaging){
app.use('/robots.txt', function (req, res) {
res.type('text/plain');
res.send("User-agent: *\nDisallow: /");
});
}
app.use(express.static(path.join(__dirname, 'public')));
Solution 8 - node.js
app.use(express.static('public'))
app.use('/images', express.static('public/images'))
app.use('/videos', express.static('public/videos'))
