2FA has its critics:

1. It’s so unreliable!
2. Phones are so easily hijacked
3. It’s not a lot of added security
4. etc

Some of these make assumptions not necessarily in evidence.

In this post:

• FreeBSD 12.1
• pam_google_authenticator-1.08

Most of the 2FA I use is time-based one-off passwords (TOTP), as opposed to text messages. These are often 6-digit numbers which change every 30 seconds. These are hard to guess and cannot be intercepted as they reside only upon your device. Sometimes they are generated by key-fob like devices. I use a smart-phone app for this.

As with all security decisions, you consider how much of a target you are and proceed accordingly.

One thing to consider: what if your device is not available? In my case, I could have used my tablet, but I did not think about that at the time.

I don’t use 2FA everywhere, but I do in critical locations, such as bastion hosts. On that host, ssh is by ssh-key and 2FA only.

What prompted this blog post was my 2FA stopped working. That is, I would ssh into the host and not get prompted for the 2FA code. I started to look into the issue and found I had no notes.

These are now my notes.

Current symptoms

When I log into the host, I do not get prompted for my 2FA code. This me logging in with my ssh-key enabled:

[dvl@nagios02:~] $ ssh -p [redacted] dan@bar.example.org
Last login: Sat Mar 14 18:09:31 2020 from knew.int.example.org
Welcome to Dan's Bastion host!

$ 

See! No 2FA request!

In /var/log/auth.log I find:

Mar 14 19:36:36 bar sshd[80383]: user dan login class  [preauth]
Mar 14 19:36:36 bar syslogd: last message repeated 2 times
Mar 14 19:36:36 bar sshd[80383]: Failed unknown for dan from 172.16.10.82 port 14761 ssh2
Mar 14 19:36:36 bar sshd[80383]: user dan login class  [preauth]
Mar 14 19:36:36 bar sshd[80383]: Accepted publickey for dan from 172.16.10.82 port 14761 ssh2: ED25519 SHA256:3HxoFP[redacted]

If I attempt to login as a non-existent user, e.g. foo, I get a 2FA prompt.

[dvl@nagios02:~] $ ssh -p [redacted] foo@bar.example.org
Verification code: 

If I enter a 2FA value, any value, I get a password prompt:

Verification code: 
Password for foo@bar.example.org:

Meanwhile, in the logs, I see this:

Mar 14 18:53:38 bar sshd[36746]: Invalid user foo from 72.16.10.82 port 12092
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd[36746]: Failed unknown for invalid user foo from 172.16.10.82 port 12092 ssh2
Mar 14 18:53:38 bar sshd[36746]: user NOUSER login class  [preauth]
Mar 14 18:53:38 bar sshd(pam_google_authenticator)[36752]: user("foo") not found
Mar 14 18:53:38 bar sshd(pam_google_authenticator)[36752]: No secret configured for user foo, asking for code anyway.
Mar 14 18:53:38 bar sshd[36746]: Postponed keyboard-interactive for invalid user foo from 72.16.10.82 port 12092 ssh2 [preauth]

After entering the verification code, for the non-existing user, the logs show:

Mar 14 18:56:51 bar sshd(pam_google_authenticator)[45747]: Dummy password supplied by PAM. Did OpenSSH 'PermitRootLogin ' or some other config block this login?
Mar 14 18:56:51 bar syslogd: last message repeated 1 times
Mar 14 18:56:51 bar sshd(pam_google_authenticator)[45747]: Invalid verification code for foo
Mar 14 18:56:51 bar sshd[45745]: Postponed keyboard-interactive/pam for invalid user foo from 172.16.10.82 port 12345 ssh2 [preauth]

After entering the password, the logs show this:

Mar 14 18:59:06 bar sshd[48999]: error: PAM: Authentication error for illegal user foo from nagios02.example.org
Mar 14 18:59:06 bar sshd[48999]: Failed keyboard-interactive/pam for invalid user foo from 172.16.10.82 port 12415 ssh2
Mar 14 18:59:06 bar sshd[48999]: error: maximum authentication attempts exceeded for invalid user foo from 172.16.10.82 port 12415 ssh2 [preauth]
Mar 14 18:59:06 bar sshd[48999]: Disconnecting invalid user foo 172.16.10.82 port 12415: Too many authentication failures [preauth]

Setting up your google authenticator for the first time

To configure 2FA on your account, you need to be logged into that host.

$ google-authenticator 

Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
  https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/dan@bar.example.org%3Fsecret%[redacted]%26issuer%foo.example.org

QR CODE DISPLAYED HERE



Your new secret key is: [redacted]
Enter code from app (-1 to skip): [redacted]
Code confirmed
Your emergency scratch codes are:
  11692891
  32439599
  78877774
  26616938
  26841961

Do you want me to update your "/usr/home/dan/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) n

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) n

Scan the QR code with your device’s camera. Then you’re good to go.

This part was incredibly easy.

Save those codes! You’ll need them one day.

Getting the configuration correct

I made a lot of changes and the last change which fixed this for me was commenting out this in from /etc/pam.d/sshd:

auth            required        pam_unix.so             no_warn try_first_pass

Here is what I have there:

# cat /etc/pam.d/sshd
#
# $FreeBSD: releng/12.1/lib/libpam/pam.d/sshd 197769 2009-10-05 09:28:54Z des $
#
# PAM configuration for the "sshd" service
#

# auth
#auth		sufficient	pam_opie.so		no_warn no_fake_prompts
#auth		requisite	pam_opieaccess.so	no_warn allow_local
#auth		sufficient	pam_krb5.so		no_warn try_first_pass
#auth		sufficient	pam_ssh.so		no_warn try_first_pass
#auth		required	pam_unix.so		no_warn try_first_pass
auth            required        /usr/local/lib/pam_google_authenticator.so
#auth            required        pam_unix.so             no_warn try_first_pass

# account
account		required	pam_nologin.so
#account	required	pam_krb5.so
account		required	pam_login_access.so
account		required	pam_unix.so

# session
#session	optional	pam_ssh.so		want_agent
session		required	pam_permit.so

# password
#password	sufficient	pam_krb5.so		no_warn try_first_pass
#password	required	pam_unix.so		no_warn try_first_pass

And this is my simplified (all comments removed) /etc/ssh/sshd_config:

AuthorizedKeysFile	.ssh/authorized_keys
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication yes
UsePAM yes
Subsystem	sftp	/usr/libexec/sftp-server
AuthenticationMethods publickey,keyboard-interactive:pam

Hope that helps.

login without ssh-keys

If I attempt to log into this host without ssh-keys:

 $ ssh -p [redacted] dan@bar.example.org
dan@ bar.example.org: Permission denied (publickey).
$ 

These are the log entries for the above:

Mar 14 20:21:15 bar sshd[19581]: user dan login class  [preauth]
Mar 14 20:21:16 bar sshd[19581]: Connection closed by authenticating user dan 172.16.10.82 port 17502 [preauth]