Zimbra Issues

We recently upgraded to Zimbra 8.6 from 8.0.7. We hit a problem that not even zimbra support was able to figure out. Perhaps its that our Zimbra instance has been upgraded since version 3 (or 4) but it got into a bad state whenever we tried to update the proxy. Here’s the solution.

First the problem;

$ zmproxyctl status
zmnginxctl is not running
$ zmproxyctl restart
Stopping nginx...nginx is not running.
Starting nginx...nginx: [emerg] invalid port in upstream "mail.example.com:0" 
in /opt/zimbra/conf/nginx/includes/nginx.conf.web:62
failed.

So that’s fairly easy to find. 

Line 62 in conf/nginx/includes/nginx.conf.web

60 upstream zimbra_adminclient
61 {
62      server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
63      zmauth_admin;
64 }
65 # Define the collection of upstream admin console servers to which we will
66 # proxy. Define each server:port against a server directive
67 #
68 upstream zimbra_admin
69 {
70     server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
71     zmauth_admin;
72  }

The problem is the :0 

Nginx doesn’t like that as a port specification. 

Looking at the template – in this case nginx.conf.web.template

 upstream ${web.admin.upstream.name}
 {
      ${web.admin.upstream.:servers}
      zmauth_admin;
 }

So what defines that. A google gives me this;

https://wiki.zimbra.com/wiki/Zimbra_Proxy…

which says;

 NGINX Keyword:         web.admin.upstream.:servers
 Description:           List of upstream admin servers used by Web Proxy 
(i.e. admin servers for which zimbraReverseProxyLookupTarget is true, and whose
 mail mode is http|mixed|both)
 Controlling Attribute: zimbraReverseProxyLookupTarget
 Default Value:         []
 Current Value:         [<server>:7071]
 Config Text:               server   <server>:7071;
 How to modify:         N/A

Not exactly helpful. So we called Zimbra support, who suggested that we

  • Run zmproxyconfig (opt/zimbra/libexec/zmproxyconfig -e -o -w -m -C -n 7071:9071 -H `zmhostname`) – which takes down the server, as nginx fails to start. 
  • Remove all files in conf/nginx/include and re-running zmproxyconfgen – this also did not help

Digging into zmproxyconfgen arguments we find the -D flag.

-D' />,--definitions          Print variable map Definitions after loading
                          LDAP configuration (and processing
                          overrides). -D requires -s upstream server.
                          If "-s upstream server" is not specified, it
                          just dumps the default varaible map

So, diving in and looking for our problem area;

libexec/zmproxyconfgen -D -s mail.example.com 
 NGINX Keyword: web.admin.upstream.:servers
  Description: List of upstream HTTPS Admin client servers used by Web Proxy
  Value Type: CUSTOM
  Controlling Attribute: (none)
  Default Value: (none)
  Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
  Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;

So there’s our :0 again.

And elsewhere in the same output (because we know it should be 7071) we find;

NGINX Keyword: web.admin.uport
 Description: Admin console upstream port
 Value Type: INTEGER
 Controlling Attribute: zimbraAdminPort
 Default Value: 7071
 Current Value: 7071
 Config Text: 7071

We can look for zimbraAdminPort in the output and compare it to a fresh install on a test server;

1) A working test server;

$ zmprov gacf | grep AdminPort 
zimbraAdminPort: 7071
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort

2) Our problem server

$ zmprov -l gacf | grep  AdminPort 
zimbraAdminPort: 7071

In other words, we’re missing the zimbraReverseProxyAdminPortAttribute. 

Are there any other attributes we’re missing?

On the clean install;

$ zmprov gacf | grep Attribute 
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort

On the problem server

[zimbra@co1mail01 ~]$ zmprov gacf | grep Attribute 
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort

So, it looks like its just that one. 

Fixing it using;

$ zmprov -l mcf zimbraReverseProxyAdminPortAttribute zimbraAdminPort
$ zmprov -l gacf zimbraReverseProxyAdminPortAttribute
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort

Regenerating config and checking;

$ libexec/zmproxyconfgen -D -s mail.example.com > /tmp/zmproxyconfgen.out2
$ diff -u /tmp/zmproxyconfgen.out /tmp/zmproxyconfgen.out2 
--- /tmp/zmproxyconfgen.out 2016-02-09 12:30:42.856618559 +0000
+++ /tmp/zmproxyconfgen.out2 2016-02-09 12:42:37.146391380 +0000
@@ -607,8 +607,8 @@
 Value Type: CUSTOM
 Controlling Attribute: (none)
 Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
 
NGINX Keyword: web.admin.upstream.:servers
@@ -616,8 +616,8 @@
 Value Type: CUSTOM
 Controlling Attribute: (none)
 Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;

now, looking again in conf/nginx/includes

upstream zimbra_adminclient
 {
    server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
    zmauth_admin;
 }
 # Define the collection of upstream admin console servers to which we will
 # proxy. Define each server:port against a server directive
 #
 upstream zimbra_admin
 {
     server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
     zmauth_admin;
 }

Problem fixed. nginx now starts. 

All we need to do now is figure out why imap completely fails when we have the proxy active. 

Hope that helped.

Andy Phillips LMAX Exchange Staff Blogs    

Any opinions, news, research, analyses, prices or other information ("information") contained on this Blog, constitutes marketing communication and it has not been prepared in accordance with legal requirements designed to promote the independence of investment research. Further, the information contained within this Blog does not contain (and should not be construed as containing) investment advice or an investment recommendation, or an offer of, or solicitation for, a transaction in any financial instrument. LMAX Group has not verified the accuracy or basis-in-fact of any claim or statement made by any third parties as comments for every Blog entry.

LMAX Group will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information. No representation or warranty is given as to the accuracy or completeness of the above information. While the produced information was obtained from sources deemed to be reliable, LMAX Group does not provide any guarantees about the reliability of such sources. Consequently any person acting on it does so entirely at his or her own risk. It is not a place to slander, use unacceptable language or to promote LMAX Group or any other FX and CFD provider and any such postings, excessive or unjust comments and attacks will not be allowed and will be removed from the site immediately.