Zimbra Issues

2016-02-29
4 min read

We recently upgraded to Zimbra 8.6 from 8.0.7. We hit a problem that not even zimbra support was able to figure out. Perhaps its that our Zimbra instance has been upgraded since version 3 (or 4) but it got into a bad state whenever we tried to update the proxy. Here’s the solution.

First the problem;

$ zmproxyctl status
zmnginxctl is not running
$ zmproxyctl restart
Stopping nginx...nginx is not running.
Starting nginx...nginx: [emerg] invalid port in upstream "mail.example.com:0" 
in /opt/zimbra/conf/nginx/includes/nginx.conf.web:62
failed.

So that’s fairly easy to find. 

Line 62 in conf/nginx/includes/nginx.conf.web

60 upstream zimbra_adminclient
61 {
62      server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
63      zmauth_admin;
64 }
65 # Define the collection of upstream admin console servers to which we will
66 # proxy. Define each server:port against a server directive
67 #
68 upstream zimbra_admin
69 {
70     server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
71     zmauth_admin;
72  }

The problem is the :0 

Nginx doesn’t like that as a port specification. 

Looking at the template - in this case nginx.conf.web.template

 upstream ${web.admin.upstream.name}
 {
      ${web.admin.upstream.:servers}
      zmauth_admin;
 }

So what defines that. A google gives me this;

https://wiki.zimbra.com/wiki/Zimbra_Proxy…

which says;

 NGINX Keyword:         web.admin.upstream.:servers
 Description:           List of upstream admin servers used by Web Proxy 
(i.e. admin servers for which zimbraReverseProxyLookupTarget is true, and whose
 mail mode is http|mixed|both)
 Controlling Attribute: zimbraReverseProxyLookupTarget
 Default Value:         []
 Current Value:         [<server>:7071]
 Config Text:               server   <server>:7071;
 How to modify:         N/A

Not exactly helpful. So we called Zimbra support, who suggested that we

  • Run zmproxyconfig (opt/zimbra/libexec/zmproxyconfig -e -o -w -m -C -n 7071:9071 -H `zmhostname`) - which takes down the server, as nginx fails to start. 
  • Remove all files in conf/nginx/include and re-running zmproxyconfgen - this also did not help

Digging into zmproxyconfgen arguments we find the -D flag.

-D' />,--definitions          Print variable map Definitions after loading
                          LDAP configuration (and processing
                          overrides). -D requires -s upstream server.
                          If "-s upstream server" is not specified, it
                          just dumps the default varaible map

So, diving in and looking for our problem area;

libexec/zmproxyconfgen -D -s mail.example.com 
 NGINX Keyword: web.admin.upstream.:servers
  Description: List of upstream HTTPS Admin client servers used by Web Proxy
  Value Type: CUSTOM
  Controlling Attribute: (none)
  Default Value: (none)
  Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
  Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;

So there’s our :0 again.

And elsewhere in the same output (because we know it should be 7071) we find;

NGINX Keyword: web.admin.uport
 Description: Admin console upstream port
 Value Type: INTEGER
 Controlling Attribute: zimbraAdminPort
 Default Value: 7071
 Current Value: 7071
 Config Text: 7071

We can look for zimbraAdminPort in the output and compare it to a fresh install on a test server;

1) A working test server;

$ zmprov gacf | grep AdminPort 
zimbraAdminPort: 7071
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort

2) Our problem server

$ zmprov -l gacf | grep  AdminPort 
zimbraAdminPort: 7071

In other words, we’re missing the zimbraReverseProxyAdminPortAttribute. 

Are there any other attributes we’re missing?

On the clean install;

$ zmprov gacf | grep Attribute 
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort

On the problem server

[zimbra@co1mail01 ~]$ zmprov gacf | grep Attribute 
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort

So, it looks like its just that one. 

Fixing it using;

$ zmprov -l mcf zimbraReverseProxyAdminPortAttribute zimbraAdminPort
$ zmprov -l gacf zimbraReverseProxyAdminPortAttribute
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort

Regenerating config and checking;

$ libexec/zmproxyconfgen -D -s mail.example.com > /tmp/zmproxyconfgen.out2
$ diff -u /tmp/zmproxyconfgen.out /tmp/zmproxyconfgen.out2 
--- /tmp/zmproxyconfgen.out 2016-02-09 12:30:42.856618559 +0000
+++ /tmp/zmproxyconfgen.out2 2016-02-09 12:42:37.146391380 +0000
@@ -607,8 +607,8 @@
 Value Type: CUSTOM
 Controlling Attribute: (none)
 Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
 
NGINX Keyword: web.admin.upstream.:servers
@@ -616,8 +616,8 @@
 Value Type: CUSTOM
 Controlling Attribute: (none)
 Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;

now, looking again in conf/nginx/includes

upstream zimbra_adminclient
 {
    server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
    zmauth_admin;
 }
 # Define the collection of upstream admin console servers to which we will
 # proxy. Define each server:port against a server directive
 #
 upstream zimbra_admin
 {
     server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
     zmauth_admin;
 }

Problem fixed. nginx now starts. 

All we need to do now is figure out why imap completely fails when we have the proxy active. 

Hope that helped.