FreeIPA, Kerberos, LDAP, Subversion Stack Part 4: Tighten Apache

You should have a Subversion repo to which you can successfully login and browse anywhere using a recent vintage of the popular TortoiseSVN on Windows.

The last step is to tighten Apache access a bit.  This is where LDAP really shines.

 Require valid-user
 Require ldap-attribute memberOf=cn=subversion_users,cn=groups,cn=accounts,dc=mydomain,dc=la
 Require ip
 Require ip

The LDAP stuff might be new for some, so that should get you started.

The Subversion documentation warns about using path-based authorization, but then provides instructions how to do it.  Ideally, Apache’s LDAP auth should be enough.

FreeIPA, Kerberos, LDAP, Subversion Stack Part 3: LDAP Setup

If you’ve read this far, Your site config isn’t complete and we’re still working inside the <location> directive.  My last post ended with the Kerberos configs.  Get through this post and there is one to go to use Apache’s Require directive to tighten things up.

Before you continue.

It’s going to be very helpful to be able to tail the 389DS logs in the freeIPA server.  Apache’s LDAP and Kerberos auth error logs are cryptic.  If you can check the 389DS logs, you can get an idea where the issue lies.

If you don’t have an LDAP browser, you should probably get one.  It’s very helpful in being sure your paths are set up right.  I like jxplorer.  The SSL setup is pretty straightforward, especially if you have previous experience in Java’s encryption norms.

LDAP Module Settings

This configures default caching and the very important path to the certificate authority (CA) cert.  Get the CA cert using openssl’s s_client directive.  There are tons of examples on the internet.

Start a new file.

nano /etc/httpd/conf.d/ldap.conf.
LDAPSharedCacheSize 500000
LDAPCacheEntries 1024
LDAPCacheTTL 600
LDAPOpCacheEntries 1024
LDAPOpCacheTTL 600
LDAPTrustedGlobalCert CA_BASE64 /etc/httpd/certs/ca.crt myPassword?

I’m not 100% sure what’s going on with those default settings until you get to the last one.  The last one is Certificate Authority path.  You must have this in order to initiate SSL LDAP connections.  If your certificate is encrypted with a password, the password goes on the end.

The quick and dirty way to get your CA is to initiate an ssl request using openssl and the s_client directive.  Apache supports two formats for keys, pem and der.  CA_BASE64 is “pem” format.  FreeIPA exports the PEM format and so does openssl.

Be sure Apache and only Apache can read the file.


For me, in August 2017, the handy AuthBasicProvider directives did not work as advertised.  It is probably me.  If you can get it to work, it’s a very handy way to trim and reuse directives.

Here’s the LDAP portion of the subversion site.

# AuthBasicProvider ldap1 ldap2 does not work
 AuthLDAPBindDN uid=ldapsearcher,cn=users,cn=accounts,dc=mydomain,dc=la
 AuthLDAPBindPassword Preposterous!!
 AuthLDAPURL ldaps://,cn=accounts,dc=mydomain,dc=la?krbPrincipalName?sub
 LDAPTrustedClientCert CERT_BASE64 /etc/httpd/certs/apache-host.crt

Notice that I set up a normal user without host login rights that can search.  389DS has some anonymous searching permissions set, but, certainly not enough to do group searches and little logging of the permission denial.    There are a couple of different blog posts describing another way to add an unprivileged user to the 389DS config instance on the Internet. But,  the config method didn’t work for me in 2017.


Here’s the second most important stanza in the LDAP setup.  The construction of the stanza with spaces added for clarity is ldaps:// Fully Qualified Host Name / Search Base? / Search field ?sub.

A fully qualified domain name works with SSL. The “search base” is where the client begins the search for the user. The “search field” is the attribute used to find the user name passed into Apache from the login form. The “sub” directive means to search every branch below the search base.  The question marks matter.

If you don’t get this stanza set up right, LDAP just won’t work and it won’t be obvious why if you are just checking Apache’s logs.  Here is where you need to tail the 389DS logs looking for Apache’s client search activity.  The 389DS log is busy.  You’ll need good grep-fu to pick the Apache activity out.


The Apache server’s host certificate needs to be stored inside a directory Apache can read.  How about putting it in with your CA cert?  You can get the certificate from the freeIPA website GUI under the host info.  Copy/paste and set correct permissions so only Apache can read it.

Advanced Config Wish List

I haven’t worked through the method of using the user logging into the repository Kerberos credentials to do the search. Apparently, this is possible.

FreeIPA Replication and the Invalid Credentials

I had replication go down between two freeIPA hosts with the following message in the logs.

Error (49) Problem connecting to replica - LDAP error: Invalid credentials (connection error)

If you are used to a world of passwords and keys, then the assumption is something is wrong with “the password.”

Since this is Kerberos authentication, time matters.  If the clocks between the two hosts aren’t very closely synced, then replication fails.

The short answer is:

stop service ntpd.


start service ntpd.

The long answer is:

Your NTPD configuration isn’t working.  It seems like the freeIPA installer does not check for a good ntpd setup.   I’ll have another post on a functional /etc/ntp.conf soon as I get the config details confirmed.


FreeIPA, Kerberos, LDAP, Subversion Stack Part 2: Apache Kerberos Setup

Lots to configure.  This step is only the Apache configs.  There are files needed to make the configs work.  That’s the next post.

Apache SVN Root

You obviously need a base directory.  Since CentOS ships with SELinux enabled, you have to be sure the context of the new directory is correct. If you put it under /var/www, my recollection is the selinux contexts are assigned auto magically.

mkdir /var/www/svn

ls -lhZ /var/www/svn

You need to move the default instance files to another folder and change the appropriate httpd.conf directives.  This should be easy.

mkdir /var/www/default80
mkdir /var/www/default80/cgi-bin
mkdir /var/www/default80/http
ls -lhZ /var/www/default80

Double-check your permissions!  Check your SELinux contexts!

Apache LDAP Config

To be 100% clear, Kerberos provides password authentication.  LDAP via freeIPA’s 389DS instance provides access permissions.

You can do SVN without LDAP, but that’s kind of awkward as everyone in the freeIPA domain with a valid account can login.  Controlling access with Subversion’s flat file will be kind of tough for many to administer without LDAP.

mkdir /etc/httpd/certs

Be sure to keep the permissions quite tight.  Your apache user needs read access, and that’s it.

Apache SVN Directives

There is a bunch of SSL stuff to setup including the Listen directive before you get to the VirtualHost.  Your default SSL config shipped with the package should have sensible defaults.  It should be pretty easy to get the keys and cert path adjusted.

Here is the start of the Apache SVN path.

<Location /svn>
 DAV svn
 SVNParentPath /var/www/svn
 RedirectMatch ^(/svn)$ $1/
 SVNListParentPath on

A couple of things worth mentioning.

You need all of the stanzas.  RedirectMatch directive is important.  It’s important the regex matches the Location directive.

The Auth portion of the Location directive.

AuthType Kerberos
AuthName "Domain Subversion Repositories"
Krb5Keytab /etc/httpd/keytabs/kwkla3.keytab
#Use your freeIPA domain name.
KrbServiceName HTTP
#More to follow! LDAP access comes next

The authentication directives are in place.  However, lots of things missing.  We’ll need to get files from your freeIPA server. The list of things we need in no particular order.

  • LDAP permissions directive.
  • CA certificate from the freeIPA instance.
  • A signed host certificate for the Apache host.
  • A private key for the Apache host.
  • An HTTP Kerberos principal.
  • TBD

NetworkManager and the Vanishing /etc/resolv.conf

I rebooted a Fedora 25 server to find the network interface did not come up.  Using the old “ifup eth0” returned an error.

 /etc/resolv.conf "no such file or directory"

Huh?  ls /etc/resolv.conf returns /etc/resolv.conf

It turns out NetworkManager replaces /etc/resolv.conf with a symbolic link to a NetworkManager directory.

Since NetworkManager is about as useful as lipstick on a pig with a server, it has to be removed. When you remove NetworkManager, it leaves /etc/resolv.conf as a dead symbolic link.  Which, you don’t see without ls -lh /etc/resolv.conf.

When systemd’s init tries to bring up the interface without NetworkManager, there’s no /etc/resolv.conf there to write DNS information and therefore the interface never comes up.

TL;DR The three commands below fix it.

rm /etc/resolv.conf;  ifdown eth0; ifup eth0;

FreeIPA, Kerberos, LDAP, Subversion Stack Part 1: Intro

This one took me a long while to figure out.  I’m using CentOS 7.x.  Everything will be specific to the “Redhat way” of doing things.

The bigger goal is having clear roles for hosts and a solution that can scale elegantly with good security bells and whistles. FreeIPA hosts authenticate, SVN servers host files.  As long as the subversion servers have enough disk space and RAM, this should scale elegantly.

This design can even fail over pretty elegantly as long as a second server is kept in sync with the primary.   Alternately, you can get into using some kind of shared storage and run a primary/secondary very easily.

Before you go looking for the configs, please understand the following:

  1. What this solution provides is front-end authentication to a Subversion repository.
  2. What this solution does NOT provide is permissions the flat file directive AuthzSVNAccessFile addresses.
  3. It’s uncertain to me how the authzsvnaccessfile interacts with an LDAP + Kerberos setup.  It seems like authz_svn_module and LDAP and Kerberos are not well documented.  Authz_svn_module and Kerberos + LDAP are is probably another big chunk of time to figure out.  Tons of how-tos for using Apache’s flat file as user auth.   It may be the case the subversion flat file auth needs to move to the Apache config.  It’s not terrible if that’s the case as it is only a config reload for Apache.
  4. It’s not an Active Directory connection.  It’s not a matter of making some minor changes to get it to work with Active Directory.

What the final solution will do:

  • A basic functioning Subversion server with centrally hosted users.
  • Kerberos authentication from FreeIPA host.
  • Basic LDAP permission control at login.
  • Encrypted (ssl) communication between your Apache host and FreeIPA. Kerberos handling the authentication means user passwords aren’t in the clear anywhere.
  • Encrypted (ssl) communication between subversion clients and the subversion server.

What you need before you get started:

  • Root access to the FreeIPA server to grep LDAP logs.
  • A functioning FreeIPA server with enough ports open to your Apache host that Kerberos and LDAP over SSL will work.
  • The Apache server already joined to the freeIPA server.
  • An LDAP browser already configured to login via LDAPS:/  I like jxplorer.
  • Some awareness of how Kerberos works.  I don’t pretend to know it well, but I know there’s a key table that needs reading/writing and very accurate time is very important.  I will probably have a couple of unnecessary steps because I don’t fully comprehend the Kerberos way.
  • Apache server and Subversion, LDAP, Kerberos modules enabled.  I tried the GSSAPI module and could not get it working.
  • The Certificate Authority’s certificate from the FreeIPA install.
  • The Certificate Authority’s private key.
  • A private key for the Apache server. The certificate for the host from freeIPA.
  • Port 443 available on the Apache host for Apache’s SSL process.  I confined mine to localhost.
  • Another port open and listening for your subversion server.

I think that’s enough for this post.  I’ll walk through some of those requirements in later posts.


Stay tuned!

Building and Running Freeipa 4.4.4 on Centos 7

I was trying to replicate Freeipa from a Fedora 25/Freeipa 4.4.4 to NOT Fedora 25.  How hard could it be to rebuild Freeipa 4.4.4 on Centos?  It turns out it is fiddly to get the packages built.

The following is not for the faint of heart and leaves out a bunch of build environment stuff, so this is not a good package to learn rebuilding a package.  But, the instructions  should save you a ton of time.

Download Fedora 25’s Freeipa source.  It should be in this directory:

Download mod_wsgi source from Fedora 25.  It should be in this directory.

Download dinglibs source for Fedora 25.  I found it here:

Download cmocka source for Fedora 25.  I found it here:

And so on for the following: ldns, python-astroid, pylint.  I was unable to build the sssd version that ships with Fedora 25 and a couple of the Python packages.  That means the freeipa.spec file gets modified with lower version requirements and the python package requirements get commented out and installed with pip.

There are also modifications to the spec files needed to be sure the packages don’t conflict with themselves.

The modifications to the spec files to accept a lower version of some of the packages are easy.  The build script should complain the correct version isn’t available. Figure out what your distro packages and modify the spec file accordingly.

My recollection is the general order of build was:

  1. build mod-wsgi
  2. build dinglibs
  3. build cmocka
  4. build python-astroid
  5. build pylint
  6. build ldns
  7. build free-ipa

The freeipa build will probably fail a couple of times on simple dependencies in the spec file.  Just keep adjusting the spec file and you’ll get built packages in a few tries.

Here is my freeipa.spec file.  It is not perfect as a number of the packages built complain about conflicting with themselves. freeipa.spec

If, after you get the packages built and the install goes okay AND you are running in an SELinux environment, be sure to run audit2allow -w -a after the IPA server stack is up to check for policy denials.

Do This Too!

For some reason, the replication setup script demands testing TCP port 7389 as another way to reach the LDAP server.  You need a port address translation for it.

firewall-cmd --permanent --zone=public --add-rich-rule="rule 
family="ipv4" \
source address="" \
port protocol="tcp" port="7389" accept"

firewall-cmd --permanent --zone=public --add-forward-port=port=7389:proto=tcp:toport=636:toaddr=

That’s the firewall-cmd way to forward anything showing up on TCP 7389 from the LAN and sending it to 636.  Adjust as needed.  It works for me.

Good luck!


Advanced Unbound DNS Server

Unbound is a multi-function DNS server from the BSD side of the Unix world.  It supports more features than dnsmasq, in particular, lots of security features related to DNSSEC.

Below is a configuration with no explicit DNSSEC that supports Kerberos, and LDAP.  It explicitly disables DNSSEC while acting as a local domain name server.

   access-control: allow
   cache-max-ttl: 14400
   cache-min-ttl: 900
   hide-identity: yes
   hide-version: yes
   minimal-responses: yes
   prefetch: yes
   qname-minimisation: yes
   rrset-roundrobin: yes
   use-caps-for-id: yes
   verbosity: 1

      name: "."
      forward-addr:        # Google
      forward-addr:        # Google
private-domain: "mydomain.zed"
domain-insecure: "mydomain.zed"
domain-insecure: ""
local-zone: "" nodefault
local-zone: "mydomain.zed." static
local-data: "mydomain.zed. 86400 IN NS ns1.mydomain.zed."
local-data: "mydomain.zed. 86400 IN NS ns2.mydomain.zed."
local-data: "mydomain.zed. 86400 IN SOA ns1.mydomain.zed. nobody.invalid. 1 3600 1200 604800 1080"
local-data: "mydomain.zed. 86400 IN SOA ns2.mydomain.zed. nobody.invalid. 1 3600 1200 604800 1080"
local-data: "ns1.mydomain.zed. 86400 IN A"
local-data: "ns2.mydomain.zed. 86400 IN A"
local-data: "server1.mydomain.zed. IN A"
local-data-ptr: " server1.mydomain.zed"
local-data: "server2.mydomain.zed IN A"
local-data-ptr: " server2.mydomain.zed"
local-data: "idserver.mydomain.zed IN A"
local-data-ptr: " idserver.mydomain.zed"
local-data: "_kerberos._tcp.mydomain.zed. 3600 IN SRV 0 100 88 idserver.mydomain.zed"
local-data: "_kerberos._udp.mydomain.zed. 3600 IN SRV 0 100 88 idserver.mydomain.zed"
local-data: "_kerberos-adm._tcp.mydomain.zed. 3600 IN SRV 0 100 749 idserver.mydomain.zed"
local-data: '_kerberos.mydomain.zed. TXT mydomain.zed"'
local-data: "idserver CNAME idserver.mydomain.zed"
local-data: "_kerberos-master._udp.mydomain.zed. 3600 IN SRV 0 100 749 idserver.mydomain.zed"
local-data: "_kpasswd._udp.mydomain.zed. 3600 IN SRV 0 100 749 idserver.mydomain.zed"
local-data: '_kerberos.mydomain.zed. TXT "mydomain.zed"'
local-data: "_ldap._tcp.mydomain.zed. 3600 IN SRV 0 100 389 idserver.mydomain.zed"

PAY ATTENTION to the use of single-quotes for the TXT record!

There’s an unbound package for Debian Stable (Stretch at time of writing)

Things to Know about Samba 4.x in 2017

It’s April, 2017 and Samba 3.x has been used as a domain controller for many, many years.  Samba 4.x has introduced radical new capabilities while still including the 3.x kitchen sink.

I recently successfully migrated a Windows domain controller to Samba 4.x and discovered a few very important distinctions and supported capabilities of 4.x vs. 3.x CentOS vs. Debian that a new Samba admin should know.

  1. Samba 4.x is nothing like 3.x.  4.x doesn’t need much in the way of a configuration.  Samba-tool makes an smb.conf and kerberos config for you. Use them.
  2. CENTOS  Samba 4.0 domain controller functionality is entirely unsupported.  What CENTOS 7.x supports is running Samba 4.x in 3.x mode. Samba-tool does not work therefore it isn’t provided. That’s very important to know if you are trying to set up a modern Windows domain.  The reason for the required the encryption backend isn’t in CENTOS and apparently there are no plans for it.
  3. Use Debian Testing as a virtual machine.  You get everything you need with Debian Testing.  There is full Samba 4.x support, samba-tool works like the wiki states.  I run CENTOS 7.x as the host OS for a bunch of servers. Running libvirt/kvm-qemu works great.

Things to Know Before Setting Up FreeIPA

A few notes about the things I discovered  testing FreeIPA about February, 2017.

  1. It’s NOT A Windows domain controller. This should be obvious just checking the basic docs.
  2. It conflicts with samba.  The first impulse is to deny this claim because, hey, there’s ample documentation about adding samba and some kind of connector/backend from FreeIPA to Samba.  However FreeIPA forces /etc/hosts format to be opposite Samba’s demanded order.   For example freeipa and freeipa.  I’m not sure how one gets around this.
  3. The LDAP server must be dedicated to freeIPA.  Putting other databases in the same 389 server causes some confusion on the part of the FreeIPA environment.  The systemd scripts just don’t work with more than one 389 database.
  4.  Use a virtual machine like kvm.  FreeIPA can run as an lxc-container. BUT  auditing does not work.  This is a limitation of the lxc container.
  5. /etc/hosts file parsing is quite strict.  If things don’t work while the system configures itself, check the arrangement in /etc/hosts.